har-dulcify (1/2)

Category index page 1 of 2 pages, listing 100 out of 169 entries in the category har-dulcify.

Used to extract statistics connected to the Alexa rank. So far the number of organizations per domain is extracted.

List scripts in readme

datasets.non-failed.domains.ratios.sorted.tsv

Add ratio buckets' aggregate analysis

Add any disconnect coverage column

Shorter table heading

Use only domains with internal/external requests in domain count columns

Count only distinct domains matching the filtered request set

Use local copy of prepared.disconnect.services.analysis.json

Merge branch 'is-disconnect-match'

Add isNotDisconnectMatch

Multiset Disconnect match coverage

Use normalized cumulative occurrences

Use line-buffer for faster dataset-foreach output/feedback

Add isDisconnectMatch

Rename to occurrences

Add Disconnect counter buckets

Run ratio-buckets with the other questions

Remove commented out code from ratio-buckets

Extract Disconnect's domains and organizations counts

Fewer restrictions by keeping all instead of only top 100 results

Move mixed usage in between extremes

Updated columns for super wide dataset tables

Ratio buckets for internal domain

Calculate ratio buckets for different request classification properties

Allow more than two-digit columns in tables

Keep disconnect's categories per organization

Update origin-redirects with mixed security, final is secure and multiset tsv generation

Add video types

Add audio types

Update image types

Add even more font types

Shorten table headers

Non-parallel per dataset execution

Use only a single value despite faulty disconnect input

Re-add mistakenly overwritten superdomain counter

Don't save referer values

Don't keep raw disconnect values

Rename effectlive tld to public suffic, move before classification, add same primary domain as a classification for internal domains

Expand request count output

Add public suffix coverage, rename disconnect external coverage

Add request status and mime-type coverage

Add in/secure int/ext/all resources plus mixed domain coverage

Add mixed domain scope

Calculate successful, unsuccessful, non-failed and failed origins

Run all multiset queries

Multiset disconnect's selected top domains

Shorten column names

Rename and delete sum after using it for sorting

Rename and run disconnect question with the others

Regroup disconnect's categories and organizations

Compare Disconnect's blocked organizations between datasets

Some more multiset table extractions

Forgot there was a join(...) call to put the CR in

Use TSV instead of CSV for download retries

Add TSV tools

Break out cleaning CSV headers

End lines with CRLF as per http://tools.ietf.org/html/rfc4180#section-2

Skip forcing @text output and let @csv take care of it

Reuse logic for object to csv row between header and body

Add origin redirect chain question/analysis

Add isInsecure basic classification

Pass Alexa top list as a parameter

Add is superdomain classification

Fix find path exclusion by reordering the arguments

Parallelize domain extraction a bit

Use an available shuf program

Remove unique testing/debugging

Prepare large zones (.com, .net) files

Calculate differences in failed versus non-failed downloads in subsequent datasets where failed domains have been retried.

Use a simplified progress bar when downloading third-party datasets

Simplified running a command on multiple datasets

Fix tailing a null-terminated list

Refactor to tee output, more functions

Split and clean up alexa's toplist

Fix google gtm/ga/dc check url reference

Use minimal input data level

More mime-types mapped

Sort and get top results in for public suffix, mime-type counts

Fix private-prefixes by saving a reference to the current public-suffixes, fix picking the primary-domain for distinct URLs

Prevent infinite loops caused by input data being close to chunk size, fallback to merging two and two

Add private-prefixes (as opposed to public-suffixes), aggregated as a single primary-domain