Bir kıvılcım bağlamı kullanarak bir web URL'ye nasıl erişilir

Question

TextFile yöntemini kullanarak bir WEB URL'sini textFile yöntemiyle kullanmaya çalışıyorum, ancak hata alıyorum. Muhtemelen bu doğru yol değildir. Birisi lütfen kıvılcım içeriğinden bir web URL'sine nasıl erişeceğimi söyleyin.

1.3.0 kıvılcım sürümünü kullanıyorum; Scala sürüm 2.10.4 ve Java 1.7.0_21

hduser@ubuntu:~$ spark-shell 
 
Spark assembly has been built with Hive, including Datanucleus jars on classpath 
 
Welcome to 
 
     __    
 
    // // 
 
    \ \/ \/ `/ _/ '_/ 
 
    /___/ .__/\_,_/_/ /_/\_\ version 1.3.0 
 
     /_/ 
 

 
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_21) 
 
Type in expressions to have them evaluated. 
 
Type :help for more information. 
 
Spark context available as sc. 
 
SQL context available as sqlContext. 
 

 
scala> val pagecount = sc.textFile("https://www.google.co.in/?gws_rd=ssl") 
 
pagecount: org.apache.spark.rdd.RDD[String] = https://www.google.co.in/?gws_rd=ssl MapPartitionsRDD[1] at textFile at <console>:21 
 

 
scala> pagecount.count() 
 
java.io.IOException: No FileSystem for scheme: https 
 
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1383) 
 
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) 
 
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) 
 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) 
 
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) 
 
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176) 
 
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208) 
 
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:203) 
 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) 
 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) 
 
at scala.Option.getOrElse(Option.scala:120) 
 
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) 
 
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) 
 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) 
 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) 
 
at scala.Option.getOrElse(Option.scala:120) 
 
at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) 
 
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1511) 
 
at org.apache.spark.rdd.RDD.count(RDD.scala:1006) 
 
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24) 
 
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29) 
 
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31) 
 
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) 
 
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:35) 
 
at $iwC$$iwC$$iwC.<init>(<console>:37) 
 
at $iwC$$iwC.<init>(<console>:39) 
 
at $iwC.<init>(<console>:41) 
 
at <init>(<console>:43) 
 
at .<init>(<console>:47) 
 
at .<clinit>(<console>) 
 
at .<init>(<console>:7) 
 
at .<clinit>(<console>) 
 
at $print(<console>) 
 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
 
at java.lang.reflect.Method.invoke(Method.java:601) 
 
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) 
 
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338) 
 
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) 
 
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) 
 
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) 
 
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856) 
 
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901) 
 
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813) 
 
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:656) 
 
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:664) 
 
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:669) 
 
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:996) 
 
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) 
 
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) 
 
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) 
 
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944) 
 
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) 
 
at org.apache.spark.repl.Main$.main(Main.scala:31) 
 
at org.apache.spark.repl.Main.main(Main.scala) 
 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
 
at java.lang.reflect.Method.invoke(Method.java:601) 
 
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) 
 
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) 
 
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) 
 
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) 
 
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

doğrudan textFile kullanan URL içeriği alınamıyor. textFile şudur:

HDF'ler bir metin dosyası (tüm düğümler üzerine) yerel dosya sistemi veya herhangi Hadoop-desteklenen dosya sistemini URI Görüyorsunuz

oku, HTTP/HTTPS URL değil dahil.

Önce içeriği alabilir ve ardından RDDs olarak alabilirsiniz.

val html = scala.io.Source.fromURL("https://spark.apache.org/").mkString 
val list = html.split("\n").filter(_ != "") 
val rdds = sc.parallelize(list) 
val count = rdds.filter(_.contains("Spark")).count()