xmake v2.5.9 released, improved C++20 module, and supported Nim, Keil MDK and Unity Build

xmake is a lightweight cross-platform build tool based on Lua. It uses xmake.lua to maintain project builds. Compared with makefile/CMakeLists.txt, the configuration syntax is more concise and intuitive, which is very friendly to novices and can quickly get started in a short time. , Allowing users to focus more on actual project development.

In this version, we have added a lot of heavyweight new features, such as: Nim language project build support, Keil MDK, Circle and Wasi tool chain support.

In addition, we have made major improvements to C++20 Modules, not only supporting the latest gcc-11, clang and msvc compilers, but also automatic analysis of inter-module dependencies to achieve maximum parallel compilation support.

Finally, there is a more useful feature that is Unity Build support, through which we can greatly improve the compilation speed of C++ code.

• project source code
• official document
• introductory course

New feature introduction

Nimlang project construction

Recently, we added support for the Nimlang project. For related issues, see: #1756

Create an empty project

We can use the xmake create command to create an empty project.

xmake create -l nim -t console test
xmake create -l nim -t static test
xmake create -l nim -t shared test

Console program

add_rules("mode.debug", "mode.release")

target("test")
    set_kind("binary")
    add_files("src/main.nim")

$ xmake -v
[ 33%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache -o:b
uild/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

Static library

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("static")
    add_files("src/foo.nim")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")

$ xmake -v
[ 33%]: linking.release libfoo.a
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/foo/macosx/x86_64/release/nimcache --app
:staticlib --noMain --passC:-DNimMain=NimMain_B6D5BD02 --passC:-DNimMainInner=NimMainInner_B6D5B
D02 --passC:-DNimMainModule=NimMainModule_B6D5BD02 --passC:-DPreMain=PreMain_B6D5BD02 --passC:-D
PreMainInner=PreMainInner_B6D5BD02 -o:build/macosx/x86_64/release/libfoo.a src/foo.nim
[ 66%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache --pa
ssL:-Lbuild/macosx/x86_64/release --passL:-lfoo -o:build/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

Dynamic library program

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("shared")
    add_files("src/foo.nim")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")

$ xmake -rv
[ 33%]: linking.release libfoo.dylib
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/foo/macosx/x86_64/release/nimcache --app
:lib --noMain -o:build/macosx/x86_64/release/libfoo.dylib src/foo.nim
[ 66%]: linking.release test
/usr/local/bin/nim c --opt:speed --nimcache:build/.gens/test/macosx/x86_64/release/nimcache --pa
ssL:-Lbuild/macosx/x86_64/release --passL:-lfoo -o:build/macosx/x86_64/release/test src/main.nim
[100%]: build ok!

C code mixed compilation

add_rules("mode.debug", "mode.release")

target("foo")
    set_kind("static")
    add_files("src/*.c")

target("test")
    set_kind("binary")
    add_deps("foo")
    add_files("src/main.nim")

Nimble dependency package integration

For a complete example, see: Nimble Package Example

add_rules("mode.debug", "mode.release")

add_requires("nimble::zip >0.3")

target("test")
    set_kind("binary")
    add_files("src/main.nim")
    add_packages("nimble::zip")

main.nim

import zip/zlib

echo zlibVersion()

Native dependency package integration

For a complete example, see: Native Package Example

add_rules("mode.debug", "mode.release")

add_requires("zlib")

target("test")
    set_kind("binary")
    add_files("src/main.nim")
    add_packages("zlib")

main.nim

proc zlibVersion(): cstring {.cdecl, importc}

echo zlibVersion()

Unity Build acceleration

We know that C++ code compilation speed is usually very slow, because each code file needs to parse the imported header file.

With Unity Build, we accelerate the compilation of the project by combining multiple cpp files into one. The main benefit is to reduce the repetitive work of parsing and compiling the contents of the header files contained in multiple source files. The contents of the header files are usually It accounts for most of the code in the source file after preprocessing.

Unity build also reduces the overhead caused by having a large number of small source files by reducing the number of object files created and processed by the compilation chain, and allows inter-procedural analysis and optimization across files that form a unified build task (similar to optimization during effect linking ).

It can greatly improve the compilation speed of C/C++ code, usually by 30%, but depending on the complexity of the project, the benefits it brings depend on the situation of the project.

xmake has also supported this build mode in v2.5.9. For related issues, see #1019 .

How to enable?

We provide two built-in rules to deal with Unity Build for C and C++ code respectively.

add_rules("c.unity_build")
add_rules("c++.unity_build")

Batch mode

By default, as long as the above rules are set, Unity Build in Batch mode will be enabled, that is, xmake will automatically organize and merge according to the project code files.

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")

We can additionally {batchsize = 2} parameter to the rule, which means that every two C++ files are automatically merged and compiled.

The compilation effect is roughly as follows:

$ xmake -r
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_642A245F.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_bar.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_73161A20.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_F905F036.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/unity_foo.cpp
[ 11%]: ccache compiling.release build/.gens/test/unity_build/main.c
[ 77%]: linking.release test
[100%]: build ok

Since we only enabled Unity Build for C++, the C code is still compiled one by one normally. In addition, in Unity Build mode, we can still speed up the parallel compilation as much as possible without conflicting each other.

batchsize parameter is not set, all files will be merged into one file for compilation by default.

Group mode

If the automatic merging effect of the above Batch mode is not ideal, we can also use custom grouping to manually configure which files are merged together to participate in compilation, which makes users more flexible and controllable.

target("test")
    set_kind("binary")
    add_rules("c++.unity_build", {batchsize = 0}) -- disable batch mode
    add_files("src/*.c", "src/*.cpp")
    add_files("src/foo/*.c", {unity_group = "foo"})
    add_files("src/bar/*.c", {unity_group = "bar"})

We use {unity_group = "foo"} to specify the name of each group and which files are included. The files of each group will be merged into one code file separately.

In addition, batchsize = 0 also forcibly disables the Batch mode, that is to say, if there is no unity_group grouped code file, we will still compile them separately, and will not automatically turn on automatic merging.

Batch and Group mixed mode

As long as we change the above batchsize = 0 to a non-zero value, we can let the remaining code files in the grouping mode continue to turn on the Batch mode to automatically merge and compile.

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")
    add_files("src/foo/*.c", {unity_group = "foo"})
    add_files("src/bar/*.c", {unity_group = "bar"})

Ignore specified file

If it is in Batch mode, because it is an automatic merge operation, all files will be merged by default, but if there are some code files that we don’t want to participate in the merge, we can also ignore them {unity_ignored = true}

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2})
    add_files("src/*.c", "src/*.cpp")
    add_files("src/test/*.c", {unity_ignored = true}) -- ignore these files

Unique ID

Although the benefits of Unity Build are good, we still encounter some unexpected situations. For example, in our two code files, under the global namespace, there are global variables and functions with the same name.

Then, merge compilation will bring about compilation conflicts, and the compiler usually reports global variable redefinition errors.

In order to solve this problem, we need to make some modifications to the user code, and then cooperate with the build tool to solve it.

For example, our foo.cpp and bar.cpp both have global variable i.

foo.cpp

namespace {
    int i = 42;
}

int foo()
{
    return i;
}

bar.cpp

namespace {
    int i = 42;
}

int bar()
{
    return i;
}

Then, we merge and compile will conflict, we can introduce a Unique ID to isolate the global anonymous space.

foo.cpp

namespace MY_UNITY_ID {
    int i = 42;
}

int foo()
{
    return MY_UNITY_ID::i;
}

bar.cpp

namespace MY_UNITY_ID {
    int i = 42;
}

int bar()
{
    return MY_UNITY_ID::i;
}

Next, we also need to ensure that after the code is merged, the MY_UNITY_ID in foo and bar are completely different. You can calculate a unique ID value based on the file name and do not conflict with each other, which is to achieve the following merge effect:

#define MY_UNITY_ID <hash(foo.cpp)>
#include "foo.c"
#undef MY_UNITY_ID
#define MY_UNITY_ID <hash(bar.cpp)>
#include "bar.c"
#undef MY_UNITY_ID

This may seem troublesome, but the user does not need to care about these, xmake will automatically process them when merging, the user only needs to specify the name of the Unique ID, for example, the following:

target("test")
    set_kind("binary")
    add_includedirs("src")
    add_rules("c++.unity_build", {batchsize = 2, uniqueid = "MY_UNITY_ID"})
    add_files("src/*.c", "src/*.cpp")

Dealing with global variables, as well as global macro definitions with the same name, functions, etc., can be used in this way to avoid conflicts.

C++20 Modules

xmake uses .mpp as the default module extension, but also supports extensions such as .ixx , .cppm , .mxx

In the early days, xmake experimentally supported C++ Modules TS, but at that time, gcc could not support it well, and the dependencies between modules were not supported either.

Recently, we have made a lot of improvements to xmake. It has fully supported the C++20 Modules construction support of gcc-11/clang/msvc, and can automatically analyze the dependencies between modules to maximize parallel compilation.

At the same time, the new version of clang/msvc has also been better handled.

set_languages("c++20")
target("test")
    set_kind("binary")
    add_files("src/*.cpp", "src/*.mpp")

For more examples, see: C++ Modules

Lua5.4 runtime support

In the last version, we added support for the Lua5.3 runtime. In this version, we further upgraded the Lua runtime to 5.4. Compared to 5.3, the runtime performance and memory utilization have been greatly improved.

However, the current default runtime of xmake is still luajit. It is expected that version 2.6.1 (that is, the next version) will officially switch to Lua5.4 as the default runtime.

Although the Lua runtime is switched, it is completely unaware to the user side and fully compatible with the existing project configuration, because xmake originally provides a layer of encapsulation for the exposed api.
Interfaces that have compatibility issues between Lua versions, such as setfenv, ffi, etc., are hidden internally and are not exposed to users.

Keil MDK tool chain support

In this version, we also added Keil/MDK embedded compilation tool chain support, related example projects: Example

xmake will automatically detect the compiler installed by Keil/MDK, related issues #1753 .

Compile with armcc

$ xmake f -p cross -a cortex-m3 --toolchain=armcc -c
$ xmake

Compile with armclang

$ xmake f -p cross -a cortex-m3 --toolchain=armclang -c
$ xmake

Console program

target("hello")
    add_deps("foo")
    add_rules("mdk.console")
    add_files("src/*.c", "src/*.s")
    add_defines("__EVAL", "__MICROLIB")
    add_includedirs("src/lib/cmsis")

Static library

add_rules("mode.debug", "mode.release")

target("foo")
    add_rules("mdk.static")
    add_files("src/foo/*.c")

Wasi toolchain support

We previously supported the emcc toolchain of the wasm platform to build wasm programs, and here, we added another Wasm toolchain with WASI to replace emcc.

$ xmake f -p wasm --toolchain=wasi
$ xmake

Circle toolchain support

We also added support for the circle compiler. This is a new C++20 compiler with some interesting compile-time meta-programming features. who are interested can check it out on the official website: 1617ea87dc45a1 https://www. circle-lang.org/

$ xmake f --toolchain=circle
$ xmake

gcc-8/9/10/11 specific version support

If the user additionally installs a specific version of the gcc tool chain such as gcc-11, gcc-10, the name of the local gcc program may be /usr/bin/gcc-11 .

One way is to switch by specifying the configuration one by one xmake f --cc=gcc-11 --cxx=gcc-11 --ld=g++-11

Therefore, xmake also provides a faster switching method:

$ xmake f --toolchain=gcc-11 -c
$ xmake

You only need to specify gcc-11 to quickly switch the entire gcc tool chain.

C++17/20 compiler feature detection

xmake provides check_features auxiliary interface to detect compiler features.

includes("check_features.lua")

target("test")
    set_kind("binary")
    add_files("*.c")
    add_configfiles("config.h.in")
    configvar_check_features("HAS_CONSTEXPR", "cxx_constexpr")
    configvar_check_features("HAS_CONSEXPR_AND_STATIC_ASSERT", {"cxx_constexpr", "c_static_assert"}, {languages = "c++11"})

config.h.in

${define HAS_CONSTEXPR}
${define HAS_CONSEXPR_AND_STATIC_ASSERT}

config.h

/* #undef HAS_CONSTEXPR */
#define HAS_CONSEXPR_AND_STATIC_ASSERT 1

In version 2.5.9, we have added c++17 feature detection:

Feature name
cxx_aggregate_bases
cxx_aligned_new
cxx_capture_star_this
cxx_constexpr
cxx_deduction_guides
cxx_enumerator_attributes
cxx_fold_expressions
cxx_guaranteed_copy_elision
cxx_hex_float
cxx_if_constexpr
cxx_inheriting_constructors
cxx_inline_variables
cxx_namespace_attributes
cxx_noexcept_function_type
cxx_nontype_template_args
cxx_nontype_template_parameter_auto
cxx_range_based_for
cxx_static_assert
cxx_structured_bindings
cxx_template_template_args
cxx_variadic_using

C++20 feature detection has also been added:

Feature name
cxx_aggregate_paren_init
cxx_char8_t
cxx_concepts
cxx_conditional_explicit
cxx_consteval
cxx_constexpr
cxx_constexpr_dynamic_alloc
cxx_constexpr_in_decltype
cxx_constinit
cxx_deduction_guides
cxx_designated_initializers
cxx_generic_lambdas
cxx_impl_coroutine
cxx_impl_destroying_delete
cxx_impl_three_way_comparison
cxx_init_captures
cxx_modules
cxx_nontype_template_args
cxx_using_enum

Xrepo package virtual environment management

Enter the virtual environment

The xrepo package management tool that comes with xmake can now well support package virtual machine environment management, similar to nixpkgs of nixos.

We can customize some package configurations by adding the xmake.lua file in the current directory, and then enter a specific package virtual environment.

add_requires("zlib 1.2.11")
add_requires("python 3.x", "luajit")

$ xrepo env shell
> python --version
> luajit --version

We can also configure and load the corresponding toolchain environment in xmake.lua, such as the compilation environment for loading vs.

set_toolchains("msvc")

Manage virtual environments

We can use the following command to register the specified virtual environment configuration globally to the system for quick switching.

$ xrepo env --add /tmp/base.lua

At this time, we have saved a global virtual environment called base, and we can view it through the list command.

$ xrepo env --list
/Users/ruki/.xmake/envs:
  - base
envs(1) found!

We can also delete it.

$ xrepo env --remove base

Switch global virtual environment

If we register multiple virtual environments, we can also switch them quickly.

$ xrepo env -b base shell
> python --version

Or directly load the specified virtual environment to run specific commands

$ xrepo env -b base python --version

xrepo env -b/--bind is to bind the specified virtual environment. For more details, see: #1762

Header Only target type

For target, we added a headeronly target type of 0617ea87dc4956. For this type of target program, we will not actually compile them because it has no source files to be compiled.

But it contains a list of header files, which are usually used for the installation of headeronly library projects, the generation of file lists for IDE projects, and the generation of cmake/pkgconfig import files during the installation phase.

E.g:

add_rules("mode.release", "mode.debug")

target("foo")
    set_kind("headeronly")
    add_headerfiles("src/foo.h")
    add_rules("utils.install.cmake_importfiles")
    add_rules("utils.install.pkgconfig_importfiles")

For more details see: #1747

Find the package from CMake

Now cmake is the de facto standard, so find_package provided by CMake can find a large number of libraries and modules. We fully reuse this part of cmake to expand the integration of xmake to packages.

We can find_package("cmake::xxx") , xmake will automatically generate a cmake script to call cmake's find_package to find some packages and get the bread information.

E.g:

$ xmake l find_package cmake::ZLIB
{
  links = {
    "z"
  },
  includedirs = {
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.
15.sdk/usr/include"
  },
  linkdirs = {
    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.
15.sdk/usr/lib"
  }
}
$ xmake l find_package cmake::LibXml2
{
  links = {
    "xml2"
  },
  includedirs = {
    "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2"
  },
  linkdirs = {
    "/usr/lib"
  }
}

Specify version

find_package("cmake::OpenCV", {required_version = "4.1.1"})

Specified components

find_package("cmake::Boost", {components = {"regex", "system"}})

Default switch

find_package("cmake::Boost", {components = {"regex", "system"}, presets = {Boost_USE_STATIC_LIB = true}})
set(Boost_USE_STATIC_LIB ON) -- will be used in FindBoost.cmake
find_package(Boost REQUIRED COMPONENTS regex system)

Set environment variables

find_package("cmake::OpenCV", {envs = {CMAKE_PREFIX_PATH = "xxx"}})

Specify a custom FindFoo.cmake module script directory

mydir/cmake_modules/FindFoo.cmake

find_package("cmake::Foo", {moduledirs = "mydir/cmake_modules"})

Package dependency integration

package("xxx")
    on_fetch(function (package, opt)
         return package:find_package("cmake::xxx", opt)
    end)
package_end()

add_requires("xxx")

Package dependency integration (optional component)

package("boost")
    add_configs("regex",   { description = "Enable regex.", default = false, type = "boolean"})
    on_fetch(function (package, opt)
         opt.components = {}
         if package:config("regex") then
             table.insert(opt.components, "regex")
         end
         return package:find_package("cmake::Boost", opt)
    end)
package_end()

add_requires("boost", {configs = {regex = true}})

Related issues: #1632

Add custom commands to CMakelists.txt

We have further improved the cmake generator, now you can serialize the custom scripts in the rule into a command list, and generate them to CMakelists.txt together

However, currently only the serialization of batchcmds series scripts can be supported.

rule("foo")
    after_buildcmd(function (target, batchcmds, opt)
        batchcmds:show("hello xmake!")
        batchcmds:cp("xmake.lua", "/tmp/")
        -- batchcmds:execv("echo", {"hello", "world!"})
        -- batchcmds:runv("echo", {"hello", "world!"})
    end)

target("test")
    set_kind("binary")
    add_rules("foo")
    add_files("src/*.c")

It will generate CMakelists.txt similar to the following

# ...
add_custom_command(TARGET test
    POST_BUILD
    COMMAND echo hello xmake!
    VERBATIM
)
add_custom_command(TARGET test
    POST_BUILD
    COMMAND cp xmake.lua /tmp/
    VERBATIM
)
target_sources(test PRIVATE
    src/main.c
)

However, the ADD_CUSTOM_COMMAND 's 0617ea87dc4b7b PRE_BUILD differs greatly on different generators, which cannot meet our needs, so we have done a lot of processing to support it.

Related issues: #1735

Improve installation support for NixOS

We have also improved the get.sh installation script to better support nixOS.

update content

New features

• #1736 : Support wasi-sdk tool chain
• Support Lua 5.4 runtime
• Add gcc-8, gcc-9, gcc-10, gcc-11 tool chain
• #1623 : Support find_package to find package from cmake
• #1747 : Add set_kind("headeronly") better handle headeronly library installation
• #1019 : Support Unity build
• #1438 : Add xmake l cli.amalgamate command to support code merging
• #1765 : support nim language
• #1762 : Manage and switch the specified environment configuration xrepo env
• #1767 : Support Circle compiler
• #1753 : Armcc/armclang tool chain supporting Keil/MDK
• #1774 : add table.contains api
• #1735 : Add custom commands to cmake generator
• #1781 : Improve get.sh installation script to support nixos

Improve

• #1528 : Check c++17/20 features
• #1729 : Improve C++20 modules' support for clang/gcc/msvc, support inter-module dependency compilation and parallel optimization
• #1779 : Improve ml.exe/x86, remove the built-in -Gd option