Steps for adding tests to the LLVM automated testing

From Dmz-portal

Jump to: navigation, search

Contents

New test suite

A new test suite would be a single test or a set of tests that are not related to the LLVM testing that is currently being done. Due to the complexity of creating a new test suite, individual tests should be added to an existing test suite if possible. Also, new test suites should be made general enough to allow for new variations of the tests and new tests.

  1. Open a bug in Bugzilla with instructions for running the test suite and assign it to Brendan. The instructions should include:
    • When or how often the test suite should be run
    • How to start the test suite running
    • How the test suite finds LLVM
    • External programs the test suite uses
    • Environment variables the test suite needs set
    • Example results for the test suite
  2. Put the test suite into a git repository on the dmz-portal.
  3. Brendan creates a Factory object for the test suite in buildbot with steps that:
    • Checks out the test suite from the repository
    • collect the external programs
    • set environment variables
    • run the test
    • parse the results of the test
  4. Brendan creates a Builder object that:
    • Names the test being run
    • Points at the Factory for the test
    • Points at the build slaves needed for the test
  5. Brendan manually runs the Builder to verify that it is running correctly and returning the expected results.
  6. Brendan schedules the test to be run regularly either by adding the Builder to an existing Scheduler. Or, creating a new Scheduler to start this Build. A new Scheduler may need:
    • The repository (or repositories) that will trigger the build
    • The changes to a branch in the repository that will trigger the build
    • The interval (every night at midnight, once every 4 hours, etc.) that the build should be started.
  7. Brendan will update the bug with the following information:
    • When to expect results from the test suite
    • Where to pick up the results
    • How to interpret the results

Add a new variation of an existing test suite

A test suite variation would run the same set of tests with different parameters. It will require changes to the buildbot Factory and Builder. And, it may need changes to the test suite itself. Variations could include:

Steps for getting a new variation of a test suite added to the automated testing:

  1. Open a bug in bugzilla and assign it to Brendan. The bug should include:
    • The existing test suite
    • When or how often the new variation should be run
    • The parameters that should be changed in the test suite
  2. Check out the test suite, modify it to allow for the new variation, and check in your changes. The new variation should allow the old variations to continue to run with the same results. Please do not change or break an existing variation as this may cause false positives or unexpected failures.
  3. Brendan creates a new Factory object based on the existing Factory object with the new parameters.
  4. Brendan creates a new Builder object based on a similar Builder object to run this variation of the test suite.
  5. Brendan schedules the variation to be run regularly either by adding the Builder to an existing Scheduler=. Or, creating a new Scheduler to start this Build. A new Scheduler may need:
    • The repository (or repositories) that will trigger the build
    • The changes to a branch in the repository that will trigger the build
    • The interval (every night at midnight, once every 4 hours, etc.) that the build should be started.
  6. Brendan will update the bug noting when to expect results from the new variation.

Add a new test to a existing test suite

Depending upon the test suite, this may be the simplest way to get a basic test run. However, how this is done will vary based on the test suite you're updating.

LLVM make check-all regression tests

The following is based on the detailed instructions at http://llvm.org/docs/TestingGuide.html#rtcustom. This will run as the make check-all step in the automated testing. It runs every time LLVM is built.

  1. Check out LLVM and Clang from dmz-portal
  2. Switch to a branch used for automated testing
  3. Change to the test (or tools/clang/test if you're testing Clang.
  4. Find an appropriate sub-directory for your test or add a new directory. If you add a new directory, be sure to copy in the dg.exp file from another test sub-directory.
  5. Create a new file in the sub-directory with the name of your test and an extension that will be picked up by the dg.exp file. (By default, dg.exp looks for .ll, .c and .cpp files, but check it to be sure it will pick up your test)
  6. Your test file should have a least one line with the word "RUN:" in it. This is a shell command that tells the test suite how to run your test. If the "RUN:" line is missing, your test will fail.
  7. The test suite uses the "RUN:" line to run your test. Any LLVM tool names in the "RUN:" line will be replaced with the full path to the previously build LLVM tool. Any "$test" strings will be replaced with your test file. If any command in the "RUN:" line fails, the test fails. (see the Variables and substitutions section for other variables you can use in your "RUN:" line)
  8. Run make check-all and verify that your test was run.
  9. Check your test into the repository and push it up to the server.

LLVM test suite

This is the LLVM Testing Infrastructure which contains a number of programs that can be compiled with LLVM and executed. The test suite is designed to compile the program with GCC. Execute it with specific parameters. Then, collect the output. It, then, repeats this with the LLVM tools. Once it has the output from both runs, it compares them. The test passes if the output is the same for both GCC and LLVM. The test can, also, fail if it fails to compile the program with the LLVM tools. This runs nightly and after checkins to certain branches several different variations.

  1. Check out LLVM and clang from the dmz-portal
  2. Check out the the llvm-test-suite repository from the dmz-portal machine into the projects/test-suite directory.
  3. Switch to the mips-tests branch for the llmv-test-suite repository.
  4. Configure and build LLVM as usual.
  5. Find an appropriate place for your test in the $SOURCE/projects/test-suite tree.
  6. Verify that running make will build and run your test with both GCC and LLVM.
  7. Check your test into the llvm-test-suite repository on the mips-tests branch.

RT-RK 1500 tests

RT-RK has a set of 1500 .c files that are compiled into executables and run. The .c files are first compiled with clang into a .bc file. Then, the .bc file is compiled into a .s (or .o if using the direct object emitter) file. Then, the .s file is compiled into an executable with GCC. Lastly, the executable is run under QEMU with no arguments. At each phase, the test passes if the compile or run command runs successfully.

  1. Check out the rt-rk-llvm-tests from the dmz-portal
  2. Place your test .c file in the tests-c directory.
  3. Run the ./run-tests.sh script (using the -R option to provide the path to your LLVM binaries) verifying that your .c file is compiled into a .bc, a .s file, an executable, then executed under QEMU.
  4. Check your .c test file into the repository and push it to the server.

Plumhall

This test suite tests that LLVM is conforming to the C and C++ standards. New tests should only be added by Plumhall. If necessary, changes can be made to this test suite in the plumhall repository on the dmz-portal machine. This repository is only accessible through gitosis.

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox