NoSuchMethodError while reading from google cloud storage from Dataproc using java -


this method read cloud storage file

public static string getstringobject(string bucketname, string filename) throws exception{         blobid blobid = blobid.of(bucketname, filename);         byte[] content = storage.readallbytes(blobid);         string contentstring = new string(content, utf_8);         return contentstring;     } 

when call method dev environment read file bucket, works fine. when call method dataproc cluster while running spark job, throws following error.

exception in thread "main" java.lang.nosuchmethoderror: com.google.common.util.concurrent.moreexecutors.directexecutor()ljava/util/concurrent/executor;     @ com.google.api.gax.retrying.basicretryingfuture.<init>(basicretryingfuture.java:77)     @ com.google.api.gax.retrying.directretryingexecutor.createfuture(directretryingexecutor.java:75)     @ com.google.cloud.retryhelper.runwithretries(retryhelper.java:53)     @ com.google.cloud.storage.storageimpl.readallbytes(storageimpl.java:460)` 

here parts of maven pom.xml

<dependency>             <groupid>com.google.cloud</groupid>             <artifactid>google-cloud-storage</artifactid>             <version>1.4.0</version>         </dependency>         <dependency>     <groupid>com.google.guava</groupid>     <artifactid>guava</artifactid>     <version>19.0</version> </dependency> <dependency>     <groupid>com.google.api-client</groupid>     <artifactid>google-api-client</artifactid>     <version>1.22.0</version> </dependency> 

what doing wrong here ?

when running on dataproc, find hadoop libraries on client application classpaths. allows interaction hdfs , gcs, means hadoop version of guava present.

you can find method of working around using shade in so answer


Comments

Popular posts from this blog

resizing Telegram inline keyboard -

command line - How can a Python program background itself? -

php - "cURL error 28: Resolving timed out" on Wordpress on Azure App Service on Linux -