Setting up the Environment¶

After building libc++ you must setup your environment to test libc++ using LIT.

1. Create a shortcut to the actual lit executable so that you can invoke it easily from the command line.

   $ alias lit='python path/to/llvm/utils/lit/lit.py'
   

2. Tell LIT where to find your build configuration.

   $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
   

Example Usage¶

Once you have your environment set up and you have built libc++ you can run parts of the libc++ test suite by simply running lit on a specified test or directory. For example:

$ cd path/to/src/libcxx
$ lit -sv test/std/re # Run all of the std::regex tests
$ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
$ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic

Sometimes you’ll want to change the way LIT is running the tests. Custom options can be specified using the –param=<name>=<val> flag. The most common option you’ll want to change is the standard dialect (ie -std=c++XX). By default the test suite will select the newest C++ dialect supported by the compiler and use that. However if you want to manually specify the option like so:

$ lit -sv test/std/containers # Run the tests with the newest -std
$ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03

Occasionally you’ll want to add extra compile or link flags when testing. You can do this as follows:

$ lit -sv --param=compile_flags='-Wcustom-warning'
$ lit -sv --param=link_flags='-L/custom/library/path'

Some other common examples include:

# Specify a custom compiler.
$ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std

# Enable warnings in the test suite
$ lit -sv --param=enable_warnings=true test/std

# Use UBSAN when running the tests.
$ lit -sv --param=use_sanitizer=Undefined

Command Line Options¶

To use these options you pass them on the LIT command line as –param NAME or –param NAME=VALUE. Some options have default values specified during CMake’s configuration. Passing the option on the command line will override the default.

cxx_under_test=<path/to/compiler>¶

Specify the compiler used to build the tests.

cxx_stdlib_under_test=<stdlib name>¶

Values: libc++, libstdc++

Specify the C++ standard library being tested. Unless otherwise specified libc++ is used. This option is intended to allow running the libc++ test suite against other standard library implementations.

std=<standard version>¶

Values: c++98, c++03, c++11, c++14, c++17, c++2a

Change the standard version used when building the tests.

libcxx_site_config=<path/to/lit.site.cfg>¶

Specify the site configuration to use when running the tests. This option overrides the environment variable LIBCXX_SITE_CONFIG.

cxx_headers=<path/to/headers>¶

Specify the c++ standard library headers that are tested. By default the headers in the source tree are used.

cxx_library_root=<path/to/lib/>¶

Specify the directory of the libc++ library to be tested. By default the library folder of the build directory is used. This option cannot be used when use_system_cxx_lib is provided.

cxx_runtime_root=<path/to/lib/>¶

Specify the directory of the libc++ library to use at runtime. This directory is not added to the linkers search path. This can be used to compile tests against one version of libc++ and run them using another. The default value for this option is cxx_library_root.

use_system_cxx_lib=<bool>¶

Default: False

Enable or disable testing against the installed version of libc++ library. Note: This does not use the installed headers.

use_lit_shell=<bool>¶

Enable or disable the use of LIT’s internal shell in ShTests. If the environment variable LIT_USE_INTERNAL_SHELL is present then that is used as the default value. Otherwise the default value is True on Windows and False on every other platform.

compile_flags="<list-of-args>"¶

Specify additional compile flags as a space delimited string. Note: This options should not be used to change the standard version used.

link_flags="<list-of-args>"¶

Specify additional link flags as a space delimited string.

debug_level=<level>¶

Values: 0, 1

Enable the use of debug mode. Level 0 enables assertions and level 1 enables assertions and debugging of iterator misuse.

use_sanitizer=<sanitizer name>¶

Values: Memory, MemoryWithOrigins, Address, Undefined

Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when building libc++ then that sanitizer will be used by default.

color_diagnostics¶

Enable the use of colorized compile diagnostics. If the color_diagnostics option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is present then color diagnostics will be enabled.

Environment Variables¶

LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>¶

Specify the site configuration to use when running the tests. Also see libcxx_site_config.

LIBCXX_COLOR_DIAGNOSTICS¶

If LIBCXX_COLOR_DIAGNOSTICS is defined then the test suite will attempt to use color diagnostic outputs from the compiler. Also see color_diagnostics.

Building Benchmarks¶

The benchmark tests are not built by default. The benchmarks can be built using the cxx-benchmarks target.

An example build would look like:

$ cd build
$ cmake [options] <path to libcxx sources>
$ make cxx-benchmarks

This will build all of the benchmarks under <libcxx-src>/benchmarks to be built against the just-built libc++. The compiled tests are output into build/benchmarks.

The benchmarks can also be built against the platforms native standard library using the -DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON CMake option. This is useful for comparing the performance of libc++ to other standard libraries. The compiled benchmarks are named <test>.libcxx.out if they test libc++ and <test>.native.out otherwise.

Also See:

• Building Libc++
• CMake Options

Running Benchmarks¶

The benchmarks must be run manually by the user. Currently there is no way to run them as part of the build.

For example:

$ cd build/benchmarks
$ make cxx-benchmarks
$ ./algorithms.libcxx.out # Runs all the benchmarks
$ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks

For more information about running benchmarks see Google Benchmark.