<< All versions
Skill v1.0.1
currentAutomated scan100/100benchflow-ai/skillsbench/parallel-processing
3 files
──Details
PublishedJuly 9, 2026 at 12:46 PM
Content Hashsha256:28428c421676d35e...
Git SHA44cdda48f6e8
Bump Typepatch
──Files
Files (1 file, 2.0 KB)
SKILL.md2.0 KBactive
SKILL.md · 82 lines · 2.0 KB
version: "1.0.1" name: parallel-processing description: Parallel processing with joblib for grid search and batch computations. Use when speeding up computationally intensive tasks across multiple CPU cores.
Parallel Processing with joblib
Speed up computationally intensive tasks by distributing work across multiple CPU cores.
Basic Usage
python
from joblib import Parallel, delayeddef process_item(x):"""Process a single item."""return x ** 2# Sequentialresults = [process_item(x) for x in range(100)]# Parallel (uses all available cores)results = Parallel(n_jobs=-1)(delayed(process_item)(x) for x in range(100))
Key Parameters
- n_jobs:
-1for all cores,1for sequential, or specific number - verbose:
0(silent),10(progress),50(detailed) - backend:
'loky'(CPU-bound, default) or'threading'(I/O-bound)
Grid Search Example
python
from joblib import Parallel, delayedfrom itertools import productdef evaluate_params(param_a, param_b):"""Evaluate one parameter combination."""score = expensive_computation(param_a, param_b)return {'param_a': param_a, 'param_b': param_b, 'score': score}# Define parameter gridparams = list(product([0.1, 0.5, 1.0], [10, 20, 30]))# Parallel grid searchresults = Parallel(n_jobs=-1, verbose=10)(delayed(evaluate_params)(a, b) for a, b in params)# Filter resultsresults = [r for r in results if r is not None]best = max(results, key=lambda x: x['score'])
Pre-computing Shared Data
When all tasks need the same data, pre-compute it once:
python
# Pre-compute onceshared_data = load_data()def process_with_shared(params, data):return compute(params, data)# Pass shared data to each taskresults = Parallel(n_jobs=-1)(delayed(process_with_shared)(p, shared_data)for p in param_list)
Performance Tips
- Only worth it for tasks taking >0.1s per item (overhead cost)
- Watch memory usage - each worker gets a copy of data
- Use
verbose=10to monitor progress