Home > python > Issue while connecting Amazon s3 using pySpark

Issue while connecting Amazon s3 using pySpark

March 13Hits:1

I am using Spark 1.6 version local mode. following is my code :

First Attempt:

airline = sc.textFile("s3n://mortar-example-data/airline-data") airline.take(2) 

Second Attempt:

airline = sc.textFile("s3n://myid:[email protected]/airline-data") airline.take(2) 

the above code is throwing me following error:

Py4JJavaError: An error occurred while calling o17.partitions. : java.io.IOException: No FileSystem for scheme: s3n     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 

Not sure what is missing here to connect to S3. It will be great if someone could point me out

Related Articles

Copyright (C) 2018 ceus-now.com, All Rights Reserved. webmaster#ceus-now.com 14 q. 0.430 s.