Builders and Syncing and Artists Oh My!

December 16, 2011

Let me tell you a story.

You’re at the office when someone yells “Build’s broken!” D’oh! After poking around for a few minutes you figure it out. The artist synced to latest, but the build machine is still running so they have mismatched content and executables. Everyone wasted their time and when the builder finishes in a few minutes things magically work.

Familiar tale? Fortunately there’s a simple and slick way to handle it. I’ll describe the process in terms of perforce, but the concept applies to any source control system.

Proper Syncing

The problem is that our poor artist hero synced new content and script before the matching executable was checked in. For anyone who builds their own exes there isn’t an issue.

Assume that the builder takes an hour to run. The “good” build is what the builder started with plus what the builder outputs. Any check-ins made during that hour build process should be ignored until the next build. If the artist can sync to builder_input + builder_output then they will NEVER have code, script, data, or exes that don’t work properly together.

We’re going to create a way for a user to easily sync to exactly that.

Magic Sync Button

At my last job we took things a step further and made a magic sync button. It was a giant glowing button at the top of our one-stop-shop game and tool launcher utility.

The magic sync button did a few things. First, clicking it would sync to our latest builder result. Second, it turned yellow if there was a new builder result to sync. Third, it would automatically save progress in any open editor, close the editor, sync the new editor exe, relaunch, and reopen files/levels/etc.

It worked so well that designers and artists could sync almost exclusively through the magic sync button. All tools had checkout/submit integration with perforce so p4v was needed only to resolve conflicts.

The Dirty Details

Sounds great. How do you do it? Short answer, via p4 label and p4 tag. Longer answer:

  1. Create a new label called magic_sync_label.
  2. Builder syncs to latest and records the changelist.
  3. Builder goes forth and builds creating builder_output. This takes some amount of time.
  4. Builder associates magic_sync_label with builder_input + builder_output.
  5. Builder repeats steps 2–4 until the end of time.
  6. Magic Sync Button calls ‘p4 sync to magic_sync_label’ which will forever and always sync to source + script + data + exes that work together. Hurray!

That’s not so bad! It’s honestly just a small handful of safe, non-destructive p4 calls. Let’s dig deeper.

1 — Create New Label

p4 label magic_sync_label

2a — Builder Get Latest Changelist

Get the single most recent submitted changelist. Your builder will need to capture the output and parse out the changelist which is stored as builder_input_changelist.

p4 changelists -s submitted -m 1

Output: p4 change 7238 on 2011/12/05 by some_user@some_workspace ‘words words submit notes words’

Note: Store 7238 in builder_input_changelist.

2b — Builder Sync Latest Changelist

Sync the builder to builder_input_changelist.

3 — Builder Builds and Submits

The builder can build and take as long as it wants. When it’s done and submitted find the changelist and store it as builder_output_changelist.

p4 changelists -s submited -u builder_user -m 1

Output: p4 change 7251 on 2011/12/05 by builder_user@builder_workspace ‘words words words’

Note: Store 7251 in builder_output_changelist.

4 — Builder Tags Label

This is the step that enables the magic. Assume the following directories for a simplified example:

//depot/Game/Source
//depot/Game/RawData
//depot/Game/CookedData
//depot/Game/Exes

We want to associate Source/RawData with builder_input_changelist (7238) and CookedData/Exes with builder_output_changelist (7251). Critical note, this is done with a single atomic tag command all in one line.

p4 tag -l magic_sync_label //depot/Game/Source...@7238 //depot/Game/RawData...@7238 //depot/Game/CookedData...@7251 //depot/Game/Exes...@7251

5a — Press the Magic Button

You can have a magic button, a batch file, or simply sync to label through p4v. The double label in the command is peculiar p4 syntax that preserves workspace files not part of the label.

p4 sync @magic_sync_label,@magic_sync_label

5b — Checking for Available Update

How can the magic button check to see if an update is available? The -n symbol means preview only. If text returned by the following command ends with “- file(s) up-to-date.” then the user is fully synced to the label. Any other text means there are files to update.

p4 sync -n @magic_sync_label,@magic_sync_label

Bonus Thoughts

No operation listed is destructive. If users have local changes to a file within the magic_sync_label set a standard conflict will occur that can be resolved. You can use -n on all commands for preview testing purposes.

Conclusion

Syncing to a builder produced label is awesome and I highly recommend it. It’s easy to do and in highly volatile in-development environments it can make a world of difference.

I’d like to end this post with an open question, how do users sync in your office? Have any cool tricks or setups to share? Discuss in the comments below!