io-bound-data-processing

io-bound-data-processing is a Data pipelines & ETL skill for AI agents, published by pproenca in the dot-skills repository on GitHub. Skillspace rates io-bound-data-processing C (71/100) and it passed the safety scan; the rating is based on static analysis only.

Processing, transforming, or moving datasets that may exceed RAM on a single low-compute box — covers memory discipline (streaming, generators, dtype shrinkage), I/O access patterns (sequential vs random, mmap, async), data formats (Parquet vs CSV vs JSON, predicate pushdown), chunking & batching, spill-to-disk (external merge sort, DuckDB/Polars), pipelining (bounded queues, backpressure, checkpointing), codec selection (zstd/lz4/gzip), concurrency for I/O-bound workloads (asyncio, threads, prefetch), and observability (iowait vs CPU%, rows/sec, py-spy/strace). Trigger on "process a large file", "stream this", "out-of-core", "OOM kill", "this is slow", or code with `pd.read_csv` of multi-GB files, `requests.get(...).content` on big bodies, `BytesIO` on unbounded inputs, per-row INSERTs, sequential `requests.get` loops, falling `tqdm` rates — even if I/O or memory isn't mentioned. Complement to computer-science-algorithms.

Grade

C

Quality

71 / 100

Safety

Passed

GitHub stars

180

Unproven: rated by static analysis only, not yet compared head-to-head.

This rating describes the version of the repository we crawled; it may have changed since, so go by its current files.

Skillspace indexes and rates this skill; it does not host or endorse it. Run third-party skills at your own risk.