Building multiple msbuild projects which inherit the same projects

[pieterbrandsen]

Hi, if I would do the case I listed in the title I get a issue where Build specification 'PATH\Components\X\X.csproj' is owned by two modules: 'Y' and 'Z' To resolve the issue, remove ownership for module configuration files at 'PATH\Components\Y' or at 'PATH\Components\Z

Is there any way to get around this error without putting all my other 80 projects into one fileNameEntryPoints array which still builds each project each change.

Any help would be appreciated!

    resolvers: [
        {
            kind: "MsBuild",
            root: d`Components/Y`,
            moduleName: "Y",
            fileNameEntryPoints: [r`Y.csproj`]
        },
        {
            kind: "MsBuild",
            root: d`Components/Z`,
            moduleName: "Z",
            fileNameEntryPoints: [r`Z.csproj`]
        }
     ]
});

[smera]

Hi pieterbrandsen,
Modules are a logical construct in BuildXL that can group units of work (pips). By design they define a partition of the pip space. The error you are seeing is saying that you are defining modules that overlap in terms of the work they schedule. In this case this seems to be about project X being shared (likely as a dependency) between Y and Z.
If you are trying to build a set of MSBuild projects, I wouldn't recommend splitting them into multiple modules (unless, for example, you are trying to stitch together two repositories into the same build). You can pass a list of projects to fileNameEntryPoints (or a solution file that includes them, if that's available) and put them under the same module.
Using a single module for a set of projects (vs multiple ones) doesn't have any effect in terms of cache hits, which your comment seems to imply. If you are not getting hits when you expect them, I suggest you use the cache miss analysis feature to understand the root cause.

Thanks,
Serge.

[HelcioM]

Hello everyone here! Apologies but I couldn't find any other way to contact any member of support for the usage of Microsoft(TM) \BuildXL usage with batch scripts. I have batch scripts running msbuild.exe and wondering if I could find a sample somewhere I could study?
Many thanks and kind regards, Helcio

[smera]

Hello @HelcioM,
Not sure if I completely understand your scenario. Do you have scripts orchestrated by MSBuild, or scripts that call MSBuild? For the first case, you shouldn't need to do anything specific, MSBuild should coordinate the scripts as it does with any other tool. And if you want to put BuildXL on top of MSBuild, using the MSBuild resolver should be the way to go.

[HelcioM]

Hi Serge, I hope you well and thanks a lot for taking the time replying to my email! I have interest in discovering the potential of this tool with regards to do distributed builds. I have many projects/solutions (hundreds) that I would like to build (distribute to) different VM's. My case is the second option you highlighted: scripts that call MSBuild Each of the projects are built via a call to a msbuild_build.bat file which in turn calls msbuild.exe with several parameters. Would I have to convert all of this into a MSbuild resolver similar to What this WIKI shows: BuildXL/Documentation/Wiki/Frontends/MSBuild/Main.md at main · microsoft/BuildXL (github.com)<https://github.com/microsoft/BuildXL/blob/main/Documentation/Wiki/Frontends/MSBuild/Main.md> config({ resolvers: [ { kind: "MsBuild", root: d`.`, fileNameEntryPoints: [r`HelloWorld.csproj`], moduleName: "HelloWorld", environment: Map.empty<string, string>().addRange( ["Env1", "Val1"], ["Env2", "Val2"]) globalProperties: Map.empty<string, string>().addRange( ["Configuration", "Release"], ["Platform", "x64"]), logVerbosity: "minimal", enableBinLogTracing: true }, ] ) Apologies for my laziness, but it is not straightforward to easily skin through the documentation and form an idea if I could or not build my projects or if that would be a great investment of time and effort and that is why I am asking you before hand. Kind regards Helcio

…

________________________________ From: Serge Mera ***@***.***> Sent: 15 May 2024 17:21 To: microsoft/BuildXL ***@***.***> Cc: HelcioM ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/BuildXL] Building multiple msbuild projects which inherit the same projects (Issue #1333) Hello @HelcioM<https://github.com/HelcioM>, Not sure if I completely understand your scenario. Do you have scripts orchestrated by MSBuild, or scripts that call MSBuild? For the first case, you shouldn't need to do anything specific, MSBuild should coordinate the scripts as it does with any other tool. And if you want to put BuildXL on top of MSBuild, using the MSBuild resolver should be the way to go. — Reply to this email directly, view it on GitHub<#1333 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AODOMYZ5PTJZIUBZBJNBMSLZCOKQRAVCNFSM6AAAAABHAIPYUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJTGA3TAMJZG4>. You are receiving this because you were mentioned.

[smera]

Hello @HelcioM ,
BuildXL can accelerate MSBuild provided that you already have MSBuild orchestrating a solution/multiple projects. If you are manually orchestrating multiple calls to msbuild, I think you could 1) move all projects under the same MSBuild solution, and use the bxl MSBuild resolver or 2) write DScript to describe all the calls you have to msbuild_build.bat, specifying dependencies. Without actually knowing the particular details of your setup, the first option sounds like the more standard one to me.

Thanks,
Serge.

[HelcioM]

Thank Serge. Finally, could you confirm if I can distribute the build on different VM on our network? We don't use azure and I am not clear if support is only available on Azure accounts. Thanks, Helcio Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Serge Mera ***@***.***> Sent: Tuesday, June 11, 2024 9:09:25 PM To: microsoft/BuildXL ***@***.***> Cc: HelcioM ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/BuildXL] Building multiple msbuild projects which inherit the same projects (Issue #1333) Hello @HelcioM<https://github.com/HelcioM> , BuildXL can accelerate MSBuild provided that you already have MSBuild orchestrating a solution/multiple projects. If you are manually orchestrating multiple calls to msbuild, I think you could 1) move all projects under the same MSBuild solution, and use the bxl MSBuild resolver or 2) write DScript to describe all the calls you have to msbuild_build.bat, specifying dependencies. Without actually knowing the particular details of your setup, the first option sounds like the more standard one to me. Thanks, Serge. — Reply to this email directly, view it on GitHub<#1333 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AODOMY6WZ3RX7WDRGTB4Y7LZG5KPLAVCNFSM6AAAAABHAIPYUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRGUZDKOJYGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[smera]

You don't need to run your build in an Azure pool to get a distributed build running. But you need to have a shared cache. The shared cache is used not only for caching build results, but also for communicating file artifacts across agents. Once a shared cache is setup, your agents need access to it, and also be able to reach each other from a network perspective. For the shared cache, you can use an Azure storage account backed cache. You can follow the steps described here to set it up.

Thanks,
Serge.

[HelcioM]

Ok thanks. Is that correct to assume that there is no open source out there that could be a "builxl" remote cache? It seems that there is this API that would need to be supported by the cache? Thanks Helcio

…

________________________________ From: Serge Mera ***@***.***> Sent: 12 June 2024 22:06 To: microsoft/BuildXL ***@***.***> Cc: HelcioM ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/BuildXL] Building multiple msbuild projects which inherit the same projects (Issue #1333) You don't need to run your build in an Azure pool to get a distributed build running. But you need to have a shared cache. The shared cache is used not only for caching build results, but also for communicating file artifacts across agents. Once a shared cache is setup, your agents need access to it, and also be able to reach each other from a network perspective. For the shared cache, you can use an Azure storage account backed cache. You can follow the steps described here<https://github.com/microsoft/BuildXL/blob/main/Documentation/Wiki/shared-cache.md> to set it up. Thanks, Serge. — Reply to this email directly, view it on GitHub<#1333 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AODOMYZIU3RXSRGUEV6ZNLDZHDA63AVCNFSM6AAAAABHAIPYUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTHE3TSOBVHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[smera]

You can setup a SMB-based shared cache. It may be less performant than using a blob-based cache, but it might do the job.
Essentially you need to provide BuildXL with a cache configuration file that sets this up. Use /cacheConfigFilePath:<path to json> in order to pass such file. You can follow an example of the JSON you need here: https://dev.azure.com/mseng/Domino/_git/BuildXL.Internal?path=/Public/Src/Cache/VerticalStore/BasicFilesystem/BasicFilesystemCacheFactory.cs&version=GBmain&line=24&lineEnd=25&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents. You can configure a SMB share and point CacheRootPath to it.

[HelcioM]

Hello Serge, good day! I will try to use this tool locally and build my knowledge first than I may explore more as I understand it. MS build takes many parameters and one of them is CPU count: -maxCpuCount[:number] I am wondering where I pass this particular parameter (or others supported by MSbuild.exe) below? Is not in global properties is it? Or these parameters are not supported? config({ resolvers: [ { kind: "MsBuild", root: c:\test\., fileNameEntryPoints: [rTest.sln], globalProperties: Map.empty<string, string>().addRange( ["Configuration", "Release"], ["Platform", "x64"],        ["maxCpuCount","4"]), environment: Map.empty<string, string>().addRange( ["UseEnv", "true"], logVerbosity: "normal", moduleName: "MyProject" } ] }); Kind regards Helcio

…

________________________________ From: Serge Mera ***@***.***> Sent: 12 June 2024 22:06 To: microsoft/BuildXL ***@***.***> Cc: HelcioM ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/BuildXL] Building multiple msbuild projects which inherit the same projects (Issue #1333) You don't need to run your build in an Azure pool to get a distributed build running. But you need to have a shared cache. The shared cache is used not only for caching build results, but also for communicating file artifacts across agents. Once a shared cache is setup, your agents need access to it, and also be able to reach each other from a network perspective. For the shared cache, you can use an Azure storage account backed cache. You can follow the steps described here<https://github.com/microsoft/BuildXL/blob/main/Documentation/Wiki/shared-cache.md> to set it up. Thanks, Serge. — Reply to this email directly, view it on GitHub<#1333 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AODOMYZIU3RXSRGUEV6ZNLDZHDA63AVCNFSM6AAAAABHAIPYUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTHE3TSOBVHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[smera]

This -maxCpuCount flag controls how MSBuild handles the scheduling at execution time. But in your case, you are using BuildXL to do that, the MSBuild resolver is only used to query MSBuild for the build graph. Once MSBuild returns a build graph, the execution is handled by BuildXL, so this (and other execution-side flags) are not exposed in the resolver because they are moot. There are equivalent settings on BuildXL side, but my recommendation is to start without any particular flag at this respect - the idea is that BuildXL should be smart enough to tune the concurrency model without an explicit user interaction.

Thanks,
Serge.