java - Apache Spark executor uses more than one core despite of spark.executor.cores=1 -

-

when start apache spark 1.2.1 application on centos 6.5, receive more 100% load executors in accordance 'top' output , load average significant more amount of cores.

as result have high load on garbage collector.

  • have tried limit cores per executor spark.executor.cores=1.
  • have tried spark.cores. no effect.
  • hardware 2 x intel(r) xeon(r) cpu e5-2620 v2 @ 2.10ghz, 6 physical cores each 12 cpu cores per node.
  • deployment model yarn client.

similar ubuntu 14.04 setup 4 physical cores (intel i5) has no issue, 1 core per executor.

any idea how fix this?

application submission performed code needed properties set through system.setproperty , spark configuration , context created. done same way, possible difference spark configuration properties set per-cluster there nothing special. under ubuntu 4 cores i5 leads proper load no more 1 core used each executor. under centos 6.5 2x6 cores e5 see more 1 core used per executor. more, tried apply 4 cores i5 configuration e5 , had no success.

spark-defaults.conf file content (before spark version substitution 1.2.1):

spark.master=yarn-client spark.eventlog.enabled=true spark.eventlog.dir=hdfs:///user/spark/applicationhistory spark.yarn.historyserver.address=x.x.x.x:18088 spark.executor.memory=1650m spark.executor.cores=1 spark.cores.max=4 spark.executor.instances=15 spark.shuffle.memoryfraction=0.2 spark.storage.memoryfraction=0.02 spark.yarn.jar=hdfs:///user/spark/share/lib/${spark.version}/spark-assembly.jar

main problem here see 2 x 6 cores e5 has lower performance linear 1 i5 x 4 cores. yes, e5 older should anyway notable more powerful. , yet analysed spark history server ui on similar load on both clusters see notable more time spent gc on e5 cluster. crazy state.

ok, @ end have found:

  1. actually had wrong understanding of 'cpu load' here. more 100% means execution queue length.
  2. cpu cores allocation correct. 1 core allocated per executor demanded.
  3. based on gc profiling enabled on spark executors , job code review found root cause non-optimal copying of objects. code corrected more lean in term of memory management.

so resolution was:

when see more 100% load on cpu core under spark executor should first check job gc logs ineffective memory intensive operations or @ least reduce lifetime of objects more aggressive release of resources.


Comments

Post a Comment

Popular posts from this blog

google chrome - Developer tools - How to inspect the elements which are added momentarily (by JQuery)? -

-
i looking @ how website designed using inspector in chrome. can use firefox developer tools or firefox web developer add-on or chrome web developer extension. so website has slide-show , using jquery it. looking @ markup , css in inspector, i see 2 things constantly changing slides change: one opacity css property understandable, i'd see changing values, changes fast see read anything. second- a div added in div slides change momentarily , moment when appears short notice except div . so question is there way stop slideshow can temporarily rid of blinking sort of effect (by speedy adding , removal , changing of elements , values)? where can inspect div element gets added such short moment , , gets removed? , can see changes in value of opacity takes place? is there way stop slideshow can temporarily rid of blinking sort of effect (by speedy adding , removal , changing of elements , values)? if opacity animated css transition or animation,
Read more

angularjs - Showing an empty as first option in select tag -

-
my app showing empty select field first entry. have read other solution given , tried still not working. my controller code is: .controller('usercreatecontroller', function(user,profile,auth) { alert("hello"); var vm = this; vm.type = 'create'; profile.all() //this service fetch profiles shown in select field .success(function(data){ vm.profile = data; }); vm.userdata.profile = vm.profile[0]; //here have set default value select field and html page is: <label class="col-sm-2 control-label">profile</label> <div class="col-sm-8"> <select ng-model="user.userdata.profile" ng-options="person._id person.profilename person in user.profile"> </select> </div> hierarchy of vm.profile { "_id": 46, "profile": objectid("5516498e95b84548156db754"), "isactive": true, "password": "$2
Read more

php - Cloud9 cloud IDE and CakePHP -

-
i attempting cakephp running on cloud9, cloud based ide. unaware, cloud collaboration ide intermixes services such bitbucket. site gives sudo rights, , installed cakephp using these instructions: https://www.digitalocean.com/community/tutorials/how-to-install-cakephp-on-an-ubuntu-12-04-vps when attempt run database exported machine cakephp installed, error message "url rewriting not configured on server." attempted solve issue using official guide cakephp: http://book.cakephp.org/2.0/en/installation/url-rewriting.html but no cigar, same error message. has gotten cakephp working, or aware of cloud ide free/low cost cloud9 support cakephp? here's how got working now: sudo apt-get update sudo apt-get install php5-intl curl -ss https://getcomposer.org/installer | php -- --filename=composer ./composer create-project --prefer-dist cakephp/app this creates project in /app default. move workspace docroot: mv app/* ./ mv app/.* ./ rm -rf app then de
Read more