: GitBench

f009 — submodule_usage

← submodule_usage
Sync submodule URLs
Tests ability to sync submodule remote URLs after upstream changes. Evaluates submodule-sync workflow knowledge.
medium submodule sync url upstream
Baseline Repository

These commands set up the repo before the model sees the prompt. They define the starting file structure, staged changes, and Git history.

  1. 01 git init
  2. 02 git config user.email 'test@test.com'
  3. 03 git config user.name 'Test User'
  4. 04 echo 'App' > app.txt
  5. 05 git add app.txt
  6. 06 git commit -m 'Initial commit'
  7. 07 mkdir -p ../lib-src && cd ../lib-src && git init && git config user.email 'test@test.com' && git config user.name 'Test User' && echo 'lib code' > lib.py && git add lib.py && git commit -m 'Initial lib'
  8. 08 git clone --bare ../lib-src ../lib-bare
  9. 09 git config protocol.file.allow always
  10. 10 git -c protocol.file.allow=always submodule add ../lib-bare lib
  11. 11 git add .gitmodules lib
  12. 12 git commit -m 'Add lib submodule'
  13. 13 git config submodule.lib.url ../wrong-bare
Prompt
Sync the submodule URLs from .gitmodules to .git/config. Output ONLY the git command, nothing else.
Expected
git submodule sync
Raw Attempt Evidence
Loading raw attempt evidence…
Model Outputs (14)
deepseek/deepseek-v4-flash:high PASS 100% 82 in → 165 out (159 reasoning)
git submodule sync
deepseek/deepseek-v4-flash:none__json_schema PASS 100% 83 in → 13 out (0 reasoning)
git submodule sync
JSON Schema Structured Output
(raw) { "command": "git submodule sync" }
mistralai/devstral-2512__json_schema PASS 100% 93 in → 10 out
git submodule sync
JSON Schema Structured Output
(raw) {"command": "git submodule sync"}
nvidia/nemotron-3-nano-30b-a3b:high PASS 100% 107 in → 193 out (215 reasoning)
git submodule sync
nvidia/nemotron-3-nano-30b-a3b:high__json_schema PASS 100% 104 in → 110 out (103 reasoning)
git submodule sync
JSON Schema Structured Output
(raw) { "command": "git submodule sync" }
nvidia/nemotron-3-nano-30b-a3b:none PASS 100% 103 in → 11 out (0 reasoning)
git submodule set-url lib ../lib-bare
nvidia/nemotron-3-nano-30b-a3b:none__json_schema PASS 100% 106 in → 20 out (0 reasoning)
git config -f .git/config submodule.lib.url ../lib-bare
JSON Schema Structured Output
(raw) {"command":"git config -f .git/config submodule.lib.url ../lib-bare"}
poolside/laguna-xs.2:high PASS 100% 138 in → 319 out (313 reasoning)
git submodule sync
poolside/laguna-xs.2:high__json_schema PASS 100% 136 in → 116 out (101 reasoning)
git submodule sync
JSON Schema Structured Output
(raw) { "command": "git submodule sync" }
poolside/laguna-xs.2:none PASS 100% 141 in → 16 out (0 reasoning)
git config --file .git/config submodule.lib.url ../lib-bare
poolside/laguna-xs.2:none__json_schema PASS 100% 142 in → 20 out (0 reasoning)
git config --file .git/config submodule.lib.url ../lib-bare
JSON Schema Structured Output
(raw) {"command": "git config --file .git/config submodule.lib.url ../lib-bare"}
deepseek/deepseek-v4-flash:high__json_schema FAIL 0%
Invalid JSON. Output: gitsubmodule sync
JSON Schema Structured Output
Structured Output Error
Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
Failure: Failed to parse structured JSON response: Expecting value: line 1 column 1 (char 0)
deepseek/deepseek-v4-flash:none FAIL 0% 81 in → 10 out (0 reasoning)
```bash git submodule sync ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'contains': 'lib-bare'}]
mistralai/devstral-2512 FAIL 0% 91 in → 10 out
```bash git submodule sync ```
Failure: Failed assertions: [{'type': 'git_output', 'command': 'git config --get submodule.lib.url', 'contains': 'lib-bare'}]