Parallelize repository cloning during dev environment setup by Copilot · Pull Request #72 · wp-cli/wp-cli-dev

Copilot · 2026-04-09T08:12:51Z

Sequential git clone calls during composer install/composer update made initial dev environment setup unnecessarily slow. The refresh step already used xargs for parallelism; cloning did not.

Changes

New .maintenance/clone-all-repositories.sh: Bash script that orchestrates the full clone + refresh flow. Checks that jq is available at startup with a clear error message. Detects CPU cores via nproc (Linux) or sysctl -n hw.logicalcpu (macOS), defaulting to 4 if neither is available, and caps parallelism at 8 to avoid saturating GitHub connections or IO. The cap can be overridden via the CLONE_JOBS or WPCLI_DEV_JOBS environment variables. Fetches the repository list from the GitHub API using curl -fsS (failing fast on HTTP errors), validates the response is a JSON array (with distinct messages for rate-limit errors vs. unexpected shapes), parses with jq, filters the skip list, applies the destination map, then clones missing repos and refreshes all repos in parallel using xargs -P$CORES.
New .maintenance/clone-repository.sh: Bash helper that clones a single repository, accepting exactly <destination> and <clone_url> as arguments (rejects anything other than exactly 2 args).
Removed .maintenance/clone-all-repositories.php and .maintenance/clone-repository.php.
Updated composer.json: pre-install-cmd and pre-update-cmd now call bash .maintenance/clone-all-repositories.sh.

# Check prerequisites
if ! command -v jq &>/dev/null; then
    echo "Required command 'jq' is not installed or not available in PATH." >&2
    exit 1
fi

# Detect CPU cores, cap at 8, allow env override
CORES="${CLONE_JOBS:-${WPCLI_DEV_JOBS:-${DETECTED_CORES}}}"
[[ "${CORES}" -gt 8 ]] && CORES=8

# Fetch and validate API response
if ! RESPONSE=$(curl -fsS ... ); then exit 1; fi
if ! jq -e 'type == "array"' >/dev/null <<< "${RESPONSE}"; then exit 1; fi

# Clone and refresh in parallel, bounded by core count
printf '%s\n' "${CLONE_LIST[@]}" | xargs -n2 -P"${CORES}" bash clone-repository.sh
printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -P"${CORES}" -I% php refresh-repository.php %

Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/6509f703-3861-4c87-83a6-f0247ed1904b Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

.maintenance/clone-all-repositories.php

Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/28119b52-0b7a-445b-b43e-f3189c004e28 Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

Copilot

Pull request overview

This PR speeds up initial wp-cli-dev environment setup by replacing the previous PHP-based cloning workflow with Bash scripts that clone and refresh repositories in parallel, and wires the new script into Composer’s pre-install/update hooks.

Changes:

Replace .maintenance/clone-all-repositories.php / .maintenance/clone-repository.php with Bash equivalents.
Parallelize cloning and refreshing using xargs -P with detected CPU core count.
Update composer.json hooks to run the new Bash entrypoint.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
composer.json	Switch Composer pre-install/update to call the new Bash orchestrator.
.maintenance/clone-all-repositories.sh	New orchestration script: fetch org repos, filter/map, then parallel clone + refresh.
.maintenance/clone-repository.sh	New helper script to clone a single repo given destination + URL.
.maintenance/clone-all-repositories.php	Removed old PHP orchestrator.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T21:56:07Z

.maintenance/clone-all-repositories.sh

+# Fetch repository list from the GitHub API.
+CURL_OPTS=(-s)
+if [[ -n "${GITHUB_TOKEN:-}" ]]; then
+	CURL_OPTS+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")
+fi
+
+RESPONSE=$(curl "${CURL_OPTS[@]}" 'https://api.github.com/orgs/wp-cli/repos?per_page=100')
+
+# Detect API errors such as rate limiting.
+if echo "${RESPONSE}" | jq -e '.message' &>/dev/null; then
+	MESSAGE=$(echo "${RESPONSE}" | jq -r '.message')
+	echo "GitHub responded with: ${MESSAGE}"
+	echo "If you are running into a rate limiting issue during large events please set GITHUB_TOKEN environment variable."
+	echo "See https://github.com/settings/tokens"


RESPONSE=$(curl -s ...) doesn't fail on HTTP errors and failures inside command substitution can be easy to miss under set -e. If curl fails or returns a non-JSON body, the later jq in process substitution can fail in a way that leaves the while read loop empty and the script may still exit 0 (so composer install proceeds without cloning/refreshing). Consider using curl -fS (and checking exit status) and explicitly validating that jq is available + that the response is a JSON array (or failing fast with a clear error) before continuing.

Suggested change

# Fetch repository list from the GitHub API.

CURL_OPTS=(-s)

if [[ -n "${GITHUB_TOKEN:-}" ]]; then

CURL_OPTS+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")

fi

RESPONSE=$(curl "${CURL_OPTS[@]}" 'https://api.github.com/orgs/wp-cli/repos?per_page=100')

# Detect API errors such as rate limiting.

if echo "${RESPONSE}" | jq -e '.message' &>/dev/null; then

MESSAGE=$(echo "${RESPONSE}" | jq -r '.message')

echo "GitHub responded with: ${MESSAGE}"

echo "If you are running into a rate limiting issue during large events please set GITHUB_TOKEN environment variable."

echo "See https://github.com/settings/tokens"

if ! command -v jq &>/dev/null; then

echo "Required command 'jq' is not installed or not available in PATH." >&2

exit 1

fi

# Fetch repository list from the GitHub API.

CURL_OPTS=(-fsS)

if [[ -n "${GITHUB_TOKEN:-}" ]]; then

CURL_OPTS+=(--header "Authorization: Bearer ${GITHUB_TOKEN}")

fi

if ! RESPONSE=$(curl "${CURL_OPTS[@]}" 'https://api.github.com/orgs/wp-cli/repos?per_page=100'); then

echo "Failed to fetch repository list from the GitHub API." >&2

exit 1

fi

# Validate the response shape and detect API errors such as rate limiting.

if ! jq -e 'type == "array"' >/dev/null <<< "${RESPONSE}"; then

if jq -e '.message' >/dev/null <<< "${RESPONSE}"; then

MESSAGE=$(jq -r '.message' <<< "${RESPONSE}")

echo "GitHub responded with: ${MESSAGE}" >&2

echo "If you are running into a rate limiting issue during large events please set GITHUB_TOKEN environment variable." >&2

echo "See https://github.com/settings/tokens" >&2

else

echo "GitHub API returned an unexpected response; expected a JSON array of repositories." >&2

fi

Copilot · 2026-04-09T21:56:08Z

.maintenance/clone-all-repositories.sh

+fi
+
+if [[ ${#UPDATE_FOLDERS[@]} -gt 0 ]]; then
+	printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -n1 -P"${CORES}" -I% php "${SCRIPT_DIR}/refresh-repository.php" %


xargs -n1 is redundant when using -I% (which already implies one argument per command) and can emit warnings or behave differently across GNU/BSD xargs. Consider dropping -n1 here for portability/clarity.

Suggested change

printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -n1 -P"${CORES}" -I% php "${SCRIPT_DIR}/refresh-repository.php" %

printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -P"${CORES}" -I% php "${SCRIPT_DIR}/refresh-repository.php" %

Copilot · 2026-04-09T21:56:08Z

.maintenance/clone-all-repositories.sh

+	CORES=$(nproc)
+elif command -v sysctl &>/dev/null; then
+	CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)
+else
+	CORES=4


Parallelism is set to the full logical CPU count. On high-core machines this can spawn a very large number of concurrent git/php processes, which can hit GitHub connection limits, saturate IO, or fail with resource limits. Consider capping CORES to a reasonable max (e.g., 8/16) and/or allowing an override like CLONE_JOBS/WPCLI_DEV_JOBS.

Suggested change

CORES=$(nproc)

elif command -v sysctl &>/dev/null; then

CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)

else

CORES=4

DETECTED_CORES=$(nproc)

elif command -v sysctl &>/dev/null; then

DETECTED_CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)

else

DETECTED_CORES=4

fi

MAX_CORES=8

CORES="${CLONE_JOBS:-${WPCLI_DEV_JOBS:-${DETECTED_CORES}}}"

if ! [[ "${CORES}" =~ ^[1-9][0-9]*$ ]]; then

CORES=4

elif [[ -z "${CLONE_JOBS:-}" && -z "${WPCLI_DEV_JOBS:-}" && "${CORES}" -gt "${MAX_CORES}" ]]; then

CORES=${MAX_CORES}

Copilot · 2026-04-09T21:56:08Z

.maintenance/clone-repository.sh

+
+set -euo pipefail
+
+if [[ $# -lt 2 ]]; then


The argument check allows more than 2 args (-lt 2). If xargs ever passes unexpected extra fields (e.g., due to whitespace in a destination), the script will silently ignore them and git clone may get malformed args. Consider requiring exactly 2 args (-ne 2) and failing with usage when the input isn't well-formed.

Suggested change

if [[ $# -lt 2 ]]; then

if [[ $# -ne 2 ]]; then

Copilot · 2026-04-09T21:56:08Z

composer.json

+        "pre-install-cmd": "bash .maintenance/clone-all-repositories.sh",
+        "pre-update-cmd": "bash .maintenance/clone-all-repositories.sh",


Switching Composer hooks from PHP to bash introduces new runtime prerequisites during composer install/update (at least bash, plus curl/jq used by the script). If this repo is expected to work in environments without these tools (e.g., some Windows setups or minimal CI images), consider adding a clearer failure message in the script and/or documenting the dependencies (or keeping a PHP fallback).

Suggested change

"pre-install-cmd": "bash .maintenance/clone-all-repositories.sh",

"pre-update-cmd": "bash .maintenance/clone-all-repositories.sh",

"pre-install-cmd": "php -r \"$required = ['bash', 'curl', 'jq']; $missing = []; foreach ($required as $command) { $finder = stripos(PHP_OS, 'WIN') === 0 ? 'where' : 'command -v'; @exec($finder . ' ' . escapeshellarg($command) . ' 2>' . (stripos(PHP_OS, 'WIN') === 0 ? 'NUL' : '/dev/null'), $output, $status); if ($status !== 0) { $missing[] = $command; } } if ($missing) { fwrite(STDERR, 'Missing required tools for Composer pre-install-cmd: ' . implode(', ', $missing) . PHP_EOL . 'The .maintenance/clone-all-repositories.sh hook requires bash, curl, and jq. Please install the missing dependencies and re-run composer install/update.' . PHP_EOL); exit(1); } passthru('bash .maintenance/clone-all-repositories.sh', $exitCode); exit($exitCode);\"",

"pre-update-cmd": "php -r \"$required = ['bash', 'curl', 'jq']; $missing = []; foreach ($required as $command) { $finder = stripos(PHP_OS, 'WIN') === 0 ? 'where' : 'command -v'; @exec($finder . ' ' . escapeshellarg($command) . ' 2>' . (stripos(PHP_OS, 'WIN') === 0 ? 'NUL' : '/dev/null'), $output, $status); if ($status !== 0) { $missing[] = $command; } } if ($missing) { fwrite(STDERR, 'Missing required tools for Composer pre-update-cmd: ' . implode(', ', $missing) . PHP_EOL . 'The .maintenance/clone-all-repositories.sh hook requires bash, curl, and jq. Please install the missing dependencies and re-run composer install/update.' . PHP_EOL); exit(1); } passthru('bash .maintenance/clone-all-repositories.sh', $exitCode); exit($exitCode);\"",

swissspidy · 2026-04-09T22:03:26Z

@copilot apply changes based on the comments in this thread

…n, cap cores to 8 with env override, drop redundant -n1, exact arg count in clone helper Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/aab74529-eccb-4e94-949e-b1062afb30b2 Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

Copilot · 2026-04-09T22:10:26Z

@copilot apply changes based on the comments in this thread

Done in a7630d5. Applied all suggestions from the review thread:

Added jq availability check at the top with a clear error message
Changed curl -s → curl -fsS with explicit failure handling
Validate the API response is a JSON array before processing; differentiate between rate-limit errors and other unexpected responses
Capped auto-detected cores to 8 (max), with CLONE_JOBS/WPCLI_DEV_JOBS env vars to override
Dropped redundant -n1 from the refresh xargs call
Changed -lt 2 → -ne 2 in clone-repository.sh

Initial plan

7567837

Copilot AI assigned Copilot and swissspidy Apr 9, 2026

Copilot AI linked an issue Apr 9, 2026 that may be closed by this pull request

Parallelize repository cloning during development environment setup. #71

Open

1 task

Copilot started work on behalf of swissspidy April 9, 2026 08:12 View session

Parallelize repository cloning during development environment setup

55e87f8

Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/6509f703-3861-4c87-83a6-f0247ed1904b Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Implement parallel repository cloning during development setup~~ Parallelize repository cloning during dev environment setup Apr 9, 2026

Copilot AI requested a review from swissspidy April 9, 2026 08:31

Copilot finished work on behalf of swissspidy April 9, 2026 08:31

swissspidy requested a review from thelovekesh April 9, 2026 08:47

github-actions bot added enhancement New feature or request scope:meta state:unconfirmed labels Apr 9, 2026

thelovekesh reviewed Apr 9, 2026

View reviewed changes

.maintenance/clone-all-repositories.php Outdated Show resolved Hide resolved

Copilot started work on behalf of swissspidy April 9, 2026 11:43 View session

Convert PHP cloning scripts to bash, detect CPU cores for parallelism

94182ad

Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/28119b52-0b7a-445b-b43e-f3189c004e28 Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

Copilot finished work on behalf of swissspidy April 9, 2026 12:59

swissspidy marked this pull request as ready for review April 9, 2026 21:52

swissspidy requested a review from a team as a code owner April 9, 2026 21:52

Copilot AI review requested due to automatic review settings April 9, 2026 21:52

Copilot started reviewing on behalf of swissspidy April 9, 2026 21:53 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Copilot started work on behalf of swissspidy April 9, 2026 22:03 View session

Copilot finished work on behalf of swissspidy April 9, 2026 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize repository cloning during dev environment setup#72

Parallelize repository cloning during dev environment setup#72
Copilot wants to merge 4 commits intomainfrom
copilot/parallelize-repository-cloning

Copilot AI commented Apr 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

swissspidy commented Apr 9, 2026

Uh oh!

Copilot AI commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	printf '%s\n' "${UPDATE_FOLDERS[@]}" \| xargs -n1 -P"${CORES}" -I% php "${SCRIPT_DIR}/refresh-repository.php" %
	printf '%s\n' "${UPDATE_FOLDERS[@]}" \| xargs -P"${CORES}" -I% php "${SCRIPT_DIR}/refresh-repository.php" %

-	CORES=$(nproc)
-elif command -v sysctl &>/dev/null; then
-	CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)
-else
-	CORES=4
+	DETECTED_CORES=$(nproc)
+elif command -v sysctl &>/dev/null; then
+	DETECTED_CORES=$(sysctl -n hw.logicalcpu 2>/dev/null || echo 4)
+else
+	DETECTED_CORES=4
+fi
+MAX_CORES=8
+CORES="${CLONE_JOBS:-${WPCLI_DEV_JOBS:-${DETECTED_CORES}}}"
+if ! [[ "${CORES}" =~ ^[1-9][0-9]*$ ]]; then
+	CORES=4
+elif [[ -z "${CLONE_JOBS:-}" && -z "${WPCLI_DEV_JOBS:-}" && "${CORES}" -gt "${MAX_CORES}" ]]; then
+	CORES=${MAX_CORES}

		"pre-install-cmd": "bash .maintenance/clone-all-repositories.sh",
		"pre-update-cmd": "bash .maintenance/clone-all-repositories.sh",

Conversation

Copilot AI commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

swissspidy commented Apr 9, 2026

Uh oh!

Copilot AI commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Apr 9, 2026 •

edited

Loading