使用Maven时，不要将JAR添加到Google Cloud上的暂存文件夹中

I'm running integration tests on the cloud for the Google Cloud Dataflows that I have written; checking that they read from Pub/Sub and write to BigQuery correctly, but when using Maven (mvn clean install), the staging folder is not populated with the required JARs. The only JAR that appears is a surefirebooter.jar. As a result, I get a NoClassDefFoundError for PipelineOptions (most likely because it is the first class from a dependency that's trying to be referenced) in the Stackdriver logs, and consequently the tests fail. Since they're running on the cloud I am indeed using a DataflowRunner as opposed to a DirectRunner.

我正在云上为我编写的Google云数据流运行集成测试;检查他们是否从Pub / Sub读取并正确写入BigQuery，但是在使用Maven（mvn clean install）时，staging文件夹没有填充所需的JAR。出现的唯一JAR是surefirebooter.jar。因此，我在PipedriverOptions中得到NoClassDefFoundError（很可能是因为它是试图引用的依赖项中的第一个类），因此测试失败。由于它们在云上运行，我确实使用的是DataflowRunner而不是DirectRunner。

When I run the integration tests from my IDE they work fine; the staging folder is populated with all the JARs and all is well. Also, when I run the tests using Maven but with a DirectRunner the tests run successfully, thus my problem only occurs when using Maven and a DataflowRunner. I assume that problem therefore lies with the pom.xml file, which I have given below:

当我从IDE运行集成测试时，它们工作正常; staging文件夹中填充了所有JAR，一切都很好。此外，当我使用Maven但使用DirectRunner运行测试时，测试成功运行，因此只有在使用Maven和DataflowRunner时才会出现问题。我认为问题因此在于pom.xml文件，我在下面给出了：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>group</groupId>
    <artifactId>artifact</artifactId>
    <version>1.0-SNAPSHOT</version>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>com.google.cloud.dataflow</groupId>
            <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
            <version>2.0.0-beta3</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>1.7.25</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.25</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.0.0-M3</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>2.0.2-beta</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Anyone know why this is happening and how I may resolve it?

任何人都知道为什么会这样，以及我如何解决它？

1 个解决方案

#1

When staging files, the Dataflow runner will automatically stage the the classes available to the current class loader. I believe that surefire plays some tricks with the classloader to make the tests easier to run.

在暂存文件时，Dataflow运行器将自动暂存当前类加载器可用的类。我相信surefire在类加载器中扮演一些技巧，使测试更容易运行。

One option would be to specify filesToStage on the pipeline options, which will override the normal "detect JARs to stage from the class loader". Alternatively, look at how surefire is managing the classpath, and make sure the SDK JARs are available in the classloader the test is running in.

一种选择是在管道选项上指定filesToStage，这将覆盖正常的“检测JAR以从类加载器进行分级”。或者，查看surefire如何管理类路径，并确保SDK JAR在运行测试的类加载器中可用。

1 个解决方案

#1

更多相关文章

随机推荐