Problems with the current model¶

The #include mechanism provided by the C preprocessor is a very poor way to access the API of a library, for a number of reasons:

• Compile-time scalability: Each time a header is included, the compiler must preprocess and parse the text in that header and every header it includes, transitively. This process must be repeated for every translation unit in the application, which involves a huge amount of redundant work. In a project with N translation units and M headers included in each translation unit, the compiler is performing M x N work even though most of the M headers are shared among multiple translation units. C++ is particularly bad, because the compilation model for templates forces a huge amount of code into headers.
• Fragility: #include directives are treated as textual inclusion by the preprocessor, and are therefore subject to any active macro definitions at the time of inclusion. If any of the active macro definitions happens to collide with a name in the library, it can break the library API or cause compilation failures in the library header itself. For an extreme example, #define std "The C++ Standard" and then include a standard library header: the result is a horrific cascade of failures in the C++ Standard Library’s implementation. More subtle real-world problems occur when the headers for two different libraries interact due to macro collisions, and users are forced to reorder #include directives or introduce #undef directives to break the (unintended) dependency.
• Conventional workarounds: C programmers have adopted a number of conventions to work around the fragility of the C preprocessor model. Include guards, for example, are required for the vast majority of headers to ensure that multiple inclusion doesn’t break the compile. Macro names are written with LONG_PREFIXED_UPPERCASE_IDENTIFIERS to avoid collisions, and some library/framework developers even use __underscored names in headers to avoid collisions with “normal” names that (by convention) shouldn’t even be macros. These conventions are a barrier to entry for developers coming from non-C languages, are boilerplate for more experienced developers, and make our headers far uglier than they should be.
• Tool confusion: In a C-based language, it is hard to build tools that work well with software libraries, because the boundaries of the libraries are not clear. Which headers belong to a particular library, and in what order should those headers be included to guarantee that they compile correctly? Are the headers C, C++, Objective-C++, or one of the variants of these languages? What declarations in those headers are actually meant to be part of the API, and what declarations are present only because they had to be written as part of the header file?

Semantic import¶

Modules improve access to the API of software libraries by replacing the textual preprocessor inclusion model with a more robust, more efficient semantic model. From the user’s perspective, the code looks only slightly different, because one uses an import declaration rather than a #include preprocessor directive:

import std.io; // pseudo-code; see below for syntax discussion

However, this module import behaves quite differently from the corresponding #include <stdio.h>: when the compiler sees the module import above, it loads a binary representation of the std.io module and makes its API available to the application directly. Preprocessor definitions that precede the import declaration have no impact on the API provided by std.io, because the module itself was compiled as a separate, standalone module. Additionally, any linker flags required to use the std.io module will automatically be provided when the module is imported [1] This semantic import model addresses many of the problems of the preprocessor inclusion model:

• Compile-time scalability: The std.io module is only compiled once, and importing the module into a translation unit is a constant-time operation (independent of module system). Thus, the API of each software library is only parsed once, reducing the M x N compilation problem to an M + N problem.
• Fragility: Each module is parsed as a standalone entity, so it has a consistent preprocessor environment. This completely eliminates the need for __underscored names and similarly defensive tricks. Moreover, the current preprocessor definitions when an import declaration is encountered are ignored, so one software library can not affect how another software library is compiled, eliminating include-order dependencies.
• Tool confusion: Modules describe the API of software libraries, and tools can reason about and present a module as a representation of that API. Because modules can only be built standalone, tools can rely on the module definition to ensure that they get the complete API for the library. Moreover, modules can specify which languages they work with, so, e.g., one can not accidentally attempt to load a C++ module into a C program.

Problems modules do not solve¶

Many programming languages have a module or package system, and because of the variety of features provided by these languages it is important to define what modules do not do. In particular, all of the following are considered out-of-scope for modules:

• Rewrite the world’s code: It is not realistic to require applications or software libraries to make drastic or non-backward-compatible changes, nor is it feasible to completely eliminate headers. Modules must interoperate with existing software libraries and allow a gradual transition.
• Versioning: Modules have no notion of version information. Programmers must still rely on the existing versioning mechanisms of the underlying language (if any exist) to version software libraries.
• Namespaces: Unlike in some languages, modules do not imply any notion of namespaces. Thus, a struct declared in one module will still conflict with a struct of the same name declared in a different module, just as they would if declared in two different headers. This aspect is important for backward compatibility, because (for example) the mangled names of entities in software libraries must not change when introducing modules.
• Binary distribution of modules: Headers (particularly C++ headers) expose the full complexity of the language. Maintaining a stable binary module format across architectures, compiler versions, and compiler vendors is technically infeasible.

Objective-C Import declaration¶

Objective-C provides syntax for importing a module via an @import declaration, which imports the named module:

@import std;

The @import declaration above imports the entire contents of the std module (which would contain, e.g., the entire C or C++ standard library) and make its API available within the current translation unit. To import only part of a module, one may use dot syntax to specific a particular submodule, e.g.,

@import std.io;

Redundant import declarations are ignored, and one is free to import modules at any point within the translation unit, so long as the import declaration is at global scope.

At present, there is no C or C++ syntax for import declarations. Clang will track the modules proposal in the C++ committee. See the section Includes as imports to see how modules get imported today.

Includes as imports¶

The primary user-level feature of modules is the import operation, which provides access to the API of software libraries. However, today’s programs make extensive use of #include, and it is unrealistic to assume that all of this code will change overnight. Instead, modules automatically translate #include directives into the corresponding module import. For example, the include directive

#include <stdio.h>

will be automatically mapped to an import of the module std.io. Even with specific import syntax in the language, this particular feature is important for both adoption and backward compatibility: automatic translation of #include to import allows an application to get the benefits of modules (for all modules-enabled libraries) without any changes to the application itself. Thus, users can easily use modules with one compiler while falling back to the preprocessor-inclusion mechanism with other compilers.

While building a module, #include_next is also supported, with one caveat. The usual behavior of #include_next is to search for the specified filename in the list of include paths, starting from the path after the one in which the current file was found. Because files listed in module maps are not found through include paths, a different strategy is used for #include_next directives in such files: the list of include paths is searched for the specified header name, to find the first include path that would refer to the current file. #include_next is interpreted as if the current file had been found in that path. If this search finds a file named by a module map, the #include_next directive is translated into an import, just like for a #include directive.``

Module maps¶

The crucial link between modules and headers is described by a module map, which describes how a collection of existing headers maps on to the (logical) structure of a module. For example, one could imagine a module std covering the C standard library. Each of the C standard library headers (<stdio.h>, <stdlib.h>, <math.h>, etc.) would contribute to the std module, by placing their respective APIs into the corresponding submodule (std.io, std.lib, std.math, etc.). Having a list of the headers that are part of the std module allows the compiler to build the std module as a standalone entity, and having the mapping from header names to (sub)modules allows the automatic translation of #include directives to module imports.

Module maps are specified as separate files (each named module.modulemap) alongside the headers they describe, which allows them to be added to existing software libraries without having to change the library headers themselves (in most cases [2]). The actual Module map language is described in a later section.

One can use module maps without modules to check the integrity of the use of header files. To do this, use the -fimplicit-module-maps option instead of the -fmodules option, or use -fmodule-map-file= option to explicitly specify the module map files to load.

Compilation model¶

The binary representation of modules is automatically generated by the compiler on an as-needed basis. When a module is imported (e.g., by an #include of one of the module’s headers), the compiler will spawn a second instance of itself [3], with a fresh preprocessing context [4], to parse just the headers in that module. The resulting Abstract Syntax Tree (AST) is then persisted into the binary representation of the module that is then loaded into translation unit where the module import was encountered.

The binary representation of modules is persisted in the module cache. Imports of a module will first query the module cache and, if a binary representation of the required module is already available, will load that representation directly. Thus, a module’s headers will only be parsed once per language configuration, rather than once per translation unit that uses the module.

Modules maintain references to each of the headers that were part of the module build. If any of those headers changes, or if any of the modules on which a module depends change, then the module will be (automatically) recompiled. The process should never require any user intervention.

Command-line parameters¶

-fmodules

Enable the modules feature.

-fbuiltin-module-map

Load the Clang builtins module map file. (Equivalent to -fmodule-map-file=<resource dir>/include/module.modulemap)

-fimplicit-module-maps

Enable implicit search for module map files named module.modulemap and similar. This option is implied by -fmodules. If this is disabled with -fno-implicit-module-maps, module map files will only be loaded if they are explicitly specified via -fmodule-map-file or transitively used by another module map file.

-fmodules-cache-path=<directory>

Specify the path to the modules cache. If not provided, Clang will select a system-appropriate default.

-fno-autolink

Disable automatic linking against the libraries associated with imported modules.

-fmodules-ignore-macro=macroname

Instruct modules to ignore the named macro when selecting an appropriate module variant. Use this for macros defined on the command line that don’t affect how modules are built, to improve sharing of compiled module files.

-fmodules-prune-interval=seconds

Specify the minimum delay (in seconds) between attempts to prune the module cache. Module cache pruning attempts to clear out old, unused module files so that the module cache itself does not grow without bound. The default delay is large (604,800 seconds, or 7 days) because this is an expensive operation. Set this value to 0 to turn off pruning.

-fmodules-prune-after=seconds

Specify the minimum time (in seconds) for which a file in the module cache must be unused (according to access time) before module pruning will remove it. The default delay is large (2,678,400 seconds, or 31 days) to avoid excessive module rebuilding.

-module-file-info <module file name>

Debugging aid that prints information about a given module file (with a .pcm extension), including the language and preprocessor options that particular module variant was built with.

-fmodules-decluse

Enable checking of module use declarations.

-fmodule-name=module-id

Consider a source file as a part of the given module.

-fmodule-map-file=<file>

Load the given module map file if a header from its directory or one of its subdirectories is loaded.

-fmodules-search-all

If a symbol is not found, search modules referenced in the current module maps but not imported for symbols, so the error message can reference the module by name. Note that if the global module index has not been built before, this might take some time as it needs to build all the modules. Note that this option doesn’t apply in module builds, to avoid the recursion.

-fno-implicit-modules

All modules used by the build must be specified with -fmodule-file.

-fmodule-file=<file>

Load the given precompiled module file.

-fprebuilt-module-path=<directory>

Specify the path to the prebuilt modules. If specified, we will look for modules in this directory for a given top-level module name. We don’t need a module map for loading prebuilt modules in this directory and the compiler will not try to rebuild these modules. This can be specified multiple times.

Macros¶

The C and C++ preprocessor assumes that the input text is a single linear buffer, but with modules this is not the case. It is possible to import two modules that have conflicting definitions for a macro (or where one #defines a macro and the other #undefines it). The rules for handling macro definitions in the presence of modules are as follows:

• Each definition and undefinition of a macro is considered to be a distinct entity.
• Such entities are visible if they are from the current submodule or translation unit, or if they were exported from a submodule that has been imported.
• A #define X or #undef X directive overrides all definitions of X that are visible at the point of the directive.
• A #define or #undef directive is active if it is visible and no visible directive overrides it.
• A set of macro directives is consistent if it consists of only #undef directives, or if all #define directives in the set define the macro name to the same sequence of tokens (following the usual rules for macro redefinitions).
• If a macro name is used and the set of active directives is not consistent, the program is ill-formed. Otherwise, the (unique) meaning of the macro name is used.

For example, suppose:

• <stdio.h> defines a macro getc (and exports its #define)
• <cstdio> imports the <stdio.h> module and undefines the macro (and exports its #undef)

The #undef overrides the #define, and a source file that imports both modules in any order will not see getc defined as a macro.

Lexical structure¶

Module map files use a simplified form of the C99 lexer, with the same rules for identifiers, tokens, string literals, /* */ and // comments. The module map language has the following reserved words; all other C identifiers are valid identifiers.

config_macros export     private
conflict      framework  requires
exclude       header     textual
explicit      link       umbrella
extern        module     use

Module map file¶

A module map file consists of a series of module declarations:

module-map-file:
  module-declaration*

Within a module map file, modules are referred to by a module-id, which uses periods to separate each part of a module’s name:

module-id:
  identifier ('.' identifier)*

Module declaration¶

A module declaration describes a module, including the headers that contribute to that module, its submodules, and other aspects of the module.

module-declaration:
  explicitopt frameworkopt module module-id attributesopt '{' module-member* '}'
  extern module module-id string-literal

The module-id should consist of only a single identifier, which provides the name of the module being defined. Each module shall have a single definition.

The explicit qualifier can only be applied to a submodule, i.e., a module that is nested within another module. The contents of explicit submodules are only made available when the submodule itself was explicitly named in an import declaration or was re-exported from an imported module.

The framework qualifier specifies that this module corresponds to a Darwin-style framework. A Darwin-style framework (used primarily on Mac OS X and iOS) is contained entirely in directory Name.framework, where Name is the name of the framework (and, therefore, the name of the module). That directory has the following layout:

Name.framework/
  Modules/module.modulemap  Module map for the framework
  Headers/                  Subdirectory containing framework headers
  PrivateHeaders/           Subdirectory containing framework private headers
  Frameworks/               Subdirectory containing embedded frameworks
  Resources/                Subdirectory containing additional resources
  Name                      Symbolic link to the shared library for the framework

The system attribute specifies that the module is a system module. When a system module is rebuilt, all of the module’s headers will be considered system headers, which suppresses warnings. This is equivalent to placing #pragma GCC system_header in each of the module’s headers. The form of attributes is described in the section Attributes, below.

The extern_c attribute specifies that the module contains C code that can be used from within C++. When such a module is built for use in C++ code, all of the module’s headers will be treated as if they were contained within an implicit extern "C" block. An import for a module with this attribute can appear within an extern "C" block. No other restrictions are lifted, however: the module currently cannot be imported within an extern "C" block in a namespace.

The no_undeclared_includes attribute specifies that the module can only reach non-modular headers and headers from used modules. Since some headers could be present in more than one search path and map to different modules in each path, this mechanism helps clang to find the right header, i.e., prefer the one for the current module or in a submodule instead of the first usual match in the search paths.

Modules can have a number of different kinds of members, each of which is described below:

module-member:
  requires-declaration
  header-declaration
  umbrella-dir-declaration
  submodule-declaration
  export-declaration
  use-declaration
  link-declaration
  config-macros-declaration
  conflict-declaration

An extern module references a module defined by the module-id in a file given by the string-literal. The file can be referenced either by an absolute path or by a path relative to the current map file.

requires-declaration:
  requires feature-list

feature-list:
  feature (',' feature)*

feature:
  !opt identifier

module std {
   // C standard library...

   module vector {
     requires cplusplus
     header "vector"
   }

   module type_traits {
     requires cplusplus11
     header "type_traits"
   }
 }

header-declaration:
  privateopt textualopt header string-literal header-attrsopt
  umbrella header string-literal header-attrsopt
  exclude header string-literal header-attrsopt

header-attrs:
  '{' header-attr* '}'

header-attr:
  size integer-literal
  mtime integer-literal

module std [system] {
  textual header "assert.h"
}

umbrella-dir-declaration:
  umbrella string-literal

submodule-declaration:
  module-declaration
  inferred-submodule-declaration

inferred-submodule-declaration:
  explicitopt frameworkopt module '*' attributesopt '{' inferred-submodule-member* '}'

inferred-submodule-member:
  export '*'

module MyLib {
  umbrella "MyLib"
  explicit module * {
    export *
  }
}

module MyLib {
  explicit module A {
    header "A.h"
    export *
  }

  explicit module B {
    header "B.h"
    export *
  }
}

export-declaration:
  export wildcard-module-id

wildcard-module-id:
  identifier
  '*'
  identifier '.' wildcard-module-id

module MyLib {
  module Base {
    header "Base.h"
  }

  module Derived {
    header "Derived.h"
    export Base
  }
}

module MyLib {
  module Base {
    header "Base.h"
  }

  module Derived {
    header "Derived.h"
    export *
  }
}

use-declaration:
  use module-id

module A {
  header "a.h"
}

module B {
  header "b.h"
}

module C {
  header "c.h"
  use B
}

link-declaration:
  link frameworkopt string-literal

config-macros-declaration:
  config_macros attributesopt config-macro-listopt

config-macro-list:
  identifier (',' identifier)*

module MyLogger {
  umbrella header "MyLogger.h"
  config_macros [exhaustive] NDEBUG
}

conflict-declaration:
  conflict module-id ',' string-literal

module Conflicts {
  explicit module A {
    header "conflict_a.h"
    conflict B, "we just don't like B"
  }

  module B {
    header "conflict_b.h"
  }
}

Attributes¶

Attributes are used in a number of places in the grammar to describe specific behavior of other declarations. The format of attributes is fairly simple.

attributes:
  attribute attributesopt

attribute:
  '[' identifier ']'

Any identifier can be used as an attribute, and each declaration specifies what attributes can be applied to it.

Private Module Map Files¶

Module map files are typically named module.modulemap and live either alongside the headers they describe or in a parent directory of the headers they describe. These module maps typically describe all of the API for the library.

However, in some cases, the presence or absence of particular headers is used to distinguish between the “public” and “private” APIs of a particular library. For example, a library may contain the headers Foo.h and Foo_Private.h, providing public and private APIs, respectively. Additionally, Foo_Private.h may only be available on some versions of library, and absent in others. One cannot easily express this with a single module map file in the library:

module Foo {
  header "Foo.h"

  explicit module Private {
    header "Foo_Private.h"
  }
}

because the header Foo_Private.h won’t always be available. The module map file could be customized based on whether Foo_Private.h is available or not, but doing so requires custom build machinery.

Private module map files, which are named module.private.modulemap (or, for backward compatibility, module_private.map), allow one to augment the primary module map file with an additional submodule. For example, we would split the module map file above into two module map files:

/* module.modulemap */
module Foo {
  header "Foo.h"
}

/* module.private.modulemap */
explicit module Foo.Private {
  header "Foo_Private.h"
}

When a module.private.modulemap file is found alongside a module.modulemap file, it is loaded after the module.modulemap file. In our example library, the module.private.modulemap file would be available when Foo_Private.h is available, making it easier to split a library’s public and private APIs along header boundaries.

When writing a private module as part of a framework, it’s recommended that:

• Headers for this module are present in the PrivateHeaders framework subdirectory.
• The private module is defined as a submodule of the public framework (if there’s one), similar to how Foo.Private is defined in the example above.
• The explicit keyword should be used to guarantee that its content will only be available when the submodule itself is explicitly named (through a @import for example).