scala - Running Spark Application from Eclipse -

-

i trying develop spark application on eclipse, , debug stepping through it.

i downloaded spark source code , have added of spark sub projects(such spark-core) eclipse. now, trying develop spark application using eclipse. have installed scalaide on eclipse. created simple application based on example given in spark website. 

import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext._ import org.apache.spark.sparkconf  object simpleapp {   def main(args: array[string]) {     val logfile = "your_spark_home/readme.md" // should file on system     val conf = new sparkconf().setappname("simple application")     val sc = new sparkcontext(conf)     val logdata = sc.textfile(logfile, 2).cache()     val numas = logdata.filter(line => line.contains("a")).count()     val numbs = logdata.filter(line => line.contains("b")).count()     println("lines a: %s, lines b: %s".format(numas, numbs))   } }

to project, added spark-core project dependent project(right click -> build path -> add project). now, trying build application , run it. however, project shows has errors, don't see errors listed in problems view within eclipse, nor see lines highlighted in red. so, not sure problem is. assumption need add external jars project, not sure these jars be. error caused val conf = new sparkconf().setappname("simple application") , subsequent lines. tried removing lines, , error went away. appreciate , guidance, thanks!

it seems not using package/library manager (e.g. sbt, maven) should eliminate versioning issues. it might challenging set correct version of java, scala, spark , subsequent dependencies on own. recommend change your project maven: convert existing eclipse project maven project

personally, have experiences sbt on intellij idea (https://confluence.jetbrains.com/display/intellijidea/getting+started+with+sbt) easy set , maintain.


Comments

Post a Comment

Popular posts from this blog

google chrome - Developer tools - How to inspect the elements which are added momentarily (by JQuery)? -

-
i looking @ how website designed using inspector in chrome. can use firefox developer tools or firefox web developer add-on or chrome web developer extension. so website has slide-show , using jquery it. looking @ markup , css in inspector, i see 2 things constantly changing slides change: one opacity css property understandable, i'd see changing values, changes fast see read anything. second- a div added in div slides change momentarily , moment when appears short notice except div . so question is there way stop slideshow can temporarily rid of blinking sort of effect (by speedy adding , removal , changing of elements , values)? where can inspect div element gets added such short moment , , gets removed? , can see changes in value of opacity takes place? is there way stop slideshow can temporarily rid of blinking sort of effect (by speedy adding , removal , changing of elements , values)? if opacity animated css transition or animation,
Read more

angularjs - Showing an empty as first option in select tag -

-
my app showing empty select field first entry. have read other solution given , tried still not working. my controller code is: .controller('usercreatecontroller', function(user,profile,auth) { alert("hello"); var vm = this; vm.type = 'create'; profile.all() //this service fetch profiles shown in select field .success(function(data){ vm.profile = data; }); vm.userdata.profile = vm.profile[0]; //here have set default value select field and html page is: <label class="col-sm-2 control-label">profile</label> <div class="col-sm-8"> <select ng-model="user.userdata.profile" ng-options="person._id person.profilename person in user.profile"> </select> </div> hierarchy of vm.profile { "_id": 46, "profile": objectid("5516498e95b84548156db754"), "isactive": true, "password": "$2
Read more

php - Cloud9 cloud IDE and CakePHP -

-
i attempting cakephp running on cloud9, cloud based ide. unaware, cloud collaboration ide intermixes services such bitbucket. site gives sudo rights, , installed cakephp using these instructions: https://www.digitalocean.com/community/tutorials/how-to-install-cakephp-on-an-ubuntu-12-04-vps when attempt run database exported machine cakephp installed, error message "url rewriting not configured on server." attempted solve issue using official guide cakephp: http://book.cakephp.org/2.0/en/installation/url-rewriting.html but no cigar, same error message. has gotten cakephp working, or aware of cloud ide free/low cost cloud9 support cakephp? here's how got working now: sudo apt-get update sudo apt-get install php5-intl curl -ss https://getcomposer.org/installer | php -- --filename=composer ./composer create-project --prefer-dist cakephp/app this creates project in /app default. move workspace docroot: mv app/* ./ mv app/.* ./ rm -rf app then de
Read more