मैंने ड्र्यूड का निर्माण किया, और इसमें बैच डेटा लोड करते समय त्रुटि मिलती है, इस प्रकार है:
2016-10-19T07:06:08,656 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 12
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=13910
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13910
Total vcore-seconds taken by all map tasks=13910
Total megabyte-seconds taken by all map tasks=14243840
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
2016-10-19T07:06:08,657 ERROR [task-runner-0-priority-0] io.druid.indexer.DetermineHashedPartitionsJob - Job failed: job_1476065812127_0012
2016-10-19T07:06:08,657 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[var/druid/hadoop-tmp/wikiticker/2016-10-19T070532.438Z/9d778c51830e449f831a3966a679df11]
2016-10-19T07:06:08,682 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2016-10-19T07:05:32.450Z, type=index_hadoop, dataSource=wikiticker}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
... 7 more
Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:343) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
... 7 more
2016-10-19T07:06:08,687 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2016-10-19T07:05:32.450Z] status changed to [FAILED].
2016-10-19T07:06:08,688 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_hadoop_wikiticker_2016-10-19T07:05:32.450Z",
"status" : "FAILED",
"duration" : 33220
}
फिर के अनुसार आधिकारिक समाधान, मैंने कुछ ऑपरेशन किए, लेकिन बहुत कम उपयोग:
मेरे Hadoop संस्करण में जैक्सन के संस्करण से मेल खाने के लिए Druid का pom.xml निर्भरता संपादित करें और Druid को फिर से खोलें
Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.annotations()Ljava/lang/Iterable;एक मोटा जार बनाने के लिए maven-shade- प्लगइन का उपयोग करें
java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=utf-8 -classpath ~/druid-0.9.1.1/tmp/druid-services-0.9.1.1-selfcontained.jar:~/druid-0.9.1.1/conf/druid/_common/:~/jackson-* -Ddruid.extensions.loadList=[] io.druid.cli.Main index hadoop --coordinate "org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1" ../quickstart/wikiticker-index.json
इस स्मारक के बहाने, मुझे मिल गया:
INFO: An exception was caught and reported. Message: java.lang.NullPointerException
java.lang.NullPointerException
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
at com.google.inject.spi.Elements.getElements(Elements.java:103)
at com.google.inject.spi.Elements.getElements(Elements.java:94)
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
at com.google.inject.spi.Elements.getElements(Elements.java:103)
at com.google.inject.spi.Elements.getElements(Elements.java:94)
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
at com.google.inject.spi.Elements.getElements(Elements.java:103)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
at com.google.inject.Guice.createInjector(Guice.java:96)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367)
at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60)
at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108)
at io.druid.cli.Main.main(Main.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115)
at io.druid.cli.Main.main(Main.java:105)
2016-10-20T05:05:52,401 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!!
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
Caused by: com.google.inject.CreationException: Guice creation errors:
1) An exception was caught and reported. Message: null
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173)
2) Binding to null instances is not allowed. Use toProvider(Providers.of(null)) if this is your intended behaviour.
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:93)
3) Could not find a suitable constructor in io.druid.metadata.MetadataStorageTablesConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
at io.druid.metadata.MetadataStorageTablesConfig.class(MetadataStorageTablesConfig.java:35)
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)
3 errors
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
... 6 more
Caused by: java.lang.NullPointerException
at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
... 6 more
क्या किसी को यही समस्या है। कोई मदद कर सकता है?
उत्तर:
उत्तर № 1 के लिए 1मैं अनुक्रमण कार्य चलाने में सक्षम था। कोई छायांकित जार आवश्यक नहीं। यहाँ कदम हैं।
1) Hadoop क्लाइंट डाउनलोड करें जो आपके CDH के लिए मेल खाता है। ड्र्यूड वितरण निर्देशिका में यह कमांड चलाते हैं। इसमें Hadoop-client डाउनलोड करेगा hadoop-dependencies निर्देशिका।
./bin/pull-deps -r https://repository.cloudera.com/content/repositories/releases/ -h org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1
2) मध्यमांग विन्यास में क्लाइंट संस्करण सेट करें।
hadoopDependencyCoordinates=["org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1"]
3) मैंने Hadoop की नौकरी चलाने के लिए अनुक्रमण सेवा का उपयोग किया। JSON बनाएँ, और डालें jobProperties इस तरह।
"jobProperties": {
"mapreduce.job.classloader": "true",
"mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.,org.xerial.snappy."
}
यदि आप अभी भी कुछ त्रुटियों को देखते हैं और यह जानते हैं कि यह CDH और Druid में मौजूद है, तो आवश्यक रूप से सूची में कक्षाएं जोड़ें।
4) HTTP पोस्ट करने के लिए JSON को पोस्ट करता है