unix - Spark on Yarn memory(physical+virtual) usage -


i struggling understand how memory management works spark on yarn:

my spark-submit has

--executor-memory 48g --num-executors 2

when run top -p <pids_of_2_yarn_containers/executors> 

virt    res   %mem 51.059g 0.015t ~4    (container 1) 51.039g 0.012t ~3    (container 2)

the total memory of system 380g.

and finally, on yarn when click on each of containers page can see:

resource: 54272 memory (container 1) resource: 54272 memory (container 2)

why each of above metrics not add up? requesting 48g on each spark executor, yarn shows 54g, os reports 15gb physical memory used(res column in top) , 51g virtual memory used(virt column).

in yarn-site.xml

yarn.scheduler.minimum-allocation-mb (this value changes based on cluster ram capacity) - minimum allocation every container request @ rm, in mbs. memory requests lower won't take effect, , specified value allocated @ minimum , max container size

yarn.scheduler.maximum-allocation-mb (this value changes based on cluster ram capacity) - maximum allocation every container request @ rm, in mbs. memory requests higher won't take effect, , capped value

yarn.nodemanager.resource.memory-mb - amount of physical memory, in mb, can allocated containers.

yarn.nodemanager.vmem-pmem-ratio - virtual memory (physical + paged memory) upper limit each map , reduce task determined virtual memory ratio each yarn container allowed. set following configuration, , default value 2.1

yarn.nodemanager.resource.cpu-vcores - property controls maximum sum of cores used containers on each node.

in mapred-site.xml

mapreduce.map.memory.mb - maximum memory each map task use.

mapreduce.reduce.memory.mb - maximum memory each reduce task use.

mapreduce.map.java.opts - jvm heap size map task.

mapreduce.reduce.java.opts - jvm heap size map task.

spark settings

the --executor-memory/spark.executor.memory controls executor heap size, jvms can use memory off heap, example interned strings , direct byte buffers. value of spark.yarn.executor.memoryoverhead property added executor memory determine full memory request yarn each executor. defaults max(384, .07 * spark.executor.memory)

the --num-executors command-line flag or spark.executor.instances configuration property control number of executors requested

so can specify values these parameters mentioned above. calculate memory allocations in case.


Comments

Post a Comment

Popular posts from this blog

Sort a complex associative array in PHP -

this question has answer here: php sort array subarray value 5 answers i have associative array in php like: $newarray1 = array ( [producta] => array ( [link] => http://sanjosespartan.com/blog/products/agile-java-development/ [visitcount] => 4 ) [productd] => array ( [link] => http://sanjosespartan.com/blog/products/intro-to-go/ [visitcount] => 1 ) [productg] => array ( [link] => http://sanjosespartan.com/blog/products/node-js-for-developers/ [visitcount] => 1 ) [productb] => array ( [link] => http://sanjosespartan.com/blog/products/beginning-mysql/ [visitcount] => 1 ) [productc] => array ( [link] => http://sanjosespartan.com/blog/products/computer-networks/ [visitcount] => 1 ) ...
Read more

vb.net - How to ignore if a cell is empty nothing -

how use roscolor() ignore value (like empty cell) ? color row if value upper 5 when there nothing in cell, want ignore roscolor() apply, how? public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val integer val = quotedatagridview1.rows(i).cells(3).value if val = vbempty quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white elseif val < 5 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.red elseif val > 5 , val < 10 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.lightyellow end if next end sub you can check empty value following public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val = quotedatagridview1.rows(i).cells(3).value if isdbnull(val) or val nothing quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white ...
Read more

recursion - Can every recursive algorithm be improved with dynamic programming? -

i first year undergraduate csc student looking competitive programming. recursion involves defining , solving sub problems. understand, top down dynamic programming (dp) involves memoizing solutions sub problems reduce time complexity of algorithm. can top down dp used improve efficiency of every recursive algorithm overlapping sub problems? dp fail work , how can identify this? the short answer is: yes. however, there constraints. obvious 1 recursive calls must overlap. i.e. during execution of algorithm, recursive function must called multiple times same parameters. lets truncate recursion tree memoization. can use memoization reduce number of calls. however, reduction of calls comes price. need store results somewhere. next obvious constraint need have enough memory. comes not-so obvious constraint. memory access requires time. first need find result stored , maybe copy location. in cases, might faster let recursion calculate result instead of loading somewher...
Read more