Data engineering · daily pipeline

Which developer tools are growing?

Daily downloads of 40 SDKs from 22 SaaS products on npm and PyPI. A GitHub Actions job fetches them every morning, dbt models and tests them, and this page is rebuilt only if every blocking test passes. Code on GitHub.

Momentum: the last three weeks against the three before

Weekly downloads by category

Average downloads per day, each week. Pick a category.

What the pipeline caught

Total npm downloads per day for all 18 npm packages, as the API sent them and after cleaning.

How it works

1 · EXTRACTGitHub Actions, 07:00 UTCFetches the last 14 days from both APIs, with pauses and retries for rate limits.

2 · LANDOne CSV per source per dayCommitted to the repo. Re-running overwrites the same days, so it is safe to repeat.

3 · MODELdbt on DuckDBStaging, then a model that flags missing days and dropouts, then the marts.

4 · TESTdbt testsUniqueness, gaps, unknown packages, freshness. A failure stops the deploy.

5 · PUBLISHThis pageReads one JSON file exported from the marts and goes up on GitHub Pages.