Setting Worker Machine for Distributed Builds

[rushikeshDeveloper]

While using distributed build from Orchestrator machine we specify config file for solution to build using MsBuild like "bxl /config:config.dsc /DistributedBuildOrchestratorLocation:DTP-ABC-IND:9090 /DistributedBuildRole:Orchestrator /DistributedBuildWorker:DTP-XYZ-IND:9090 /DistributedBuildServicePort:9090" and we have set inbound & outbound rule for Port 9090. But how do we initiate worker machine like we need bxl.exe running on worker machine for distributed build. And mainly on worker machine we need command like "bxl /DistributedBuildRole:Worker /DistributedBuildOrchestratorLocation:DTP-ABC-IND:9090" but additionally it asks to specify config.dsc. If so what will we specify in config file on worker machine as worker machine just need to build and all required things are specified in config.dsc file on Orchestrator machine.

[pgunasekara]

Hi @rushikeshDeveloper, you need to specify a config.dsc file on every worker - the same config file you used on the orchestrator machine. You should have the same set of source files across each worker machine (make sure you do package restore on each machine as well if that's a step that happens outside the build).

[peter-liberman]

Is a worker then limited to only one orchestrator machine or is there a way that each machine could act as both an orchestrator and a worker? Also, is there any mechanism to orchestrate executing git commands to update the source and switch branches on the worker machines?

[rushikeshDeveloper]

Hi @pgunasekara,
Thank you for the inputs you provided. Can you provide some more information about distributed builds.
Like in issue"#1310" @mpysson mentioned that we need to implement a cache for our environment for file transfer between machines. We are not sure how to implement it.

So the things we have tried is:

1. On worker machine we have started bxl.exe from cmd using "bxl /config:config.dsc /DistributedBuildOrchestratorLocation:DTP-ABC-IND:9090 /DistributedBuildRole:Worker /DistributedBuildWorker:DTP-XYZ-IND:9090 /DistributedBuildServicePort:9090" and it shows "Connecting to the orchestrator to advertise location"
2. From orchestrator/main machine we have started "bxl /config:config.dsc /DistributedBuildOrchestratorLocation:DTP-ABC-IND:9090 /DistributedBuildRole:Orchestrator /DistributedBuildWorker:DTP-XYZ-IND:9090 /DistributedBuildServicePort:9090"
3. So connection between two machines through port 9090 for bxl.exe is established confirmed using application like currports.
4. Now even when connection is established there is no data been transferred between two machines. From "BuildXL.DistributionRpc.log" we can see:
   
5. We are using Msbuild as frontend to build a solution file and found that msbuild creates a build graph. So how is this graph sent on worker machine and how will the worker machine know what to build. As on worker machine we get error "error DX7021: Distributed build worker failed because it could not load a graph". So is this related to cache only?

So can you provide us with a sample example for distributed build on both Orchestrator & Worker machine along with cache implementation or any documentations.

[pgunasekara]

@peter-liberman, yes you are limited to a single orchestrator machine. We don't provide any mechanism to perform git operations through buildxl.

[pgunasekara]

@rushikeshDeveloper I don't have any examples to share with you, however the issue you linked is outdated now. You can try set up a cache backed by azure blob storage. Here's the documentation on how to do that: https://github.com/microsoft/BuildXL/blob/main/Documentation/Wiki/shared-cache.md