Code intrsospection into autosklearn components

[eddiebergman]

This is a first draft in an attempt to address #1429. Redone to get clean history after rebasing.

Sample output:

from autosklearn.info import classifiers

name, info = next(iter(classifiers().items()))

print(name)
"adaboost"

print(info)
ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.adaboost.AdaboostClassifier'>, name='AdaBoost Classifier', shortname='AB', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=False)

The classifiers() returns dict[str, ClassifierInfo]

classifiers()

{'adaboost': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.adaboost.AdaboostClassifier'>, name='AdaBoost Classifier', shortname='AB', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=False),
 'bernoulli_nb': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.bernoulli_nb.BernoulliNB'>, name='Bernoulli Naive Bayes classifier', shortname='BernoulliNB', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'decision_tree': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.decision_tree.DecisionTree'>, name='Decision Tree Classifier', shortname='DT', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'extra_trees': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.extra_trees.ExtraTreesClassifier'>, name='Extra Trees Classifier', shortname='ET', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'gaussian_nb': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.gaussian_nb.GaussianNB'>, name='Gaussian Naive Bayes classifier', shortname='GaussianNB', output_kind='predictions', supported_inputs=['dense', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'gradient_boosting': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.gradient_boosting.GradientBoostingClassifier'>, name='Gradient Boosting Classifier', shortname='GB', output_kind='predictions', supported_inputs=['dense', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=False),
 'k_nearest_neighbors': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.k_nearest_neighbors.KNearestNeighborsClassifier'>, name='K-Nearest Neighbor Classification', shortname='KNN', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'lda': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.lda.LDA'>, name='Linear Discriminant Analysis', shortname='LDA', output_kind='predictions', supported_inputs=['dense', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'liblinear_svc': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.liblinear_svc.LibLinear_SVC'>, name='Liblinear Support Vector Classification', shortname='Liblinear-SVC', output_kind='predictions', supported_inputs=['sparse', 'dense', 'unsigned data'], deterministic=False, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'libsvm_svc': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.libsvm_svc.LibSVM_SVC'>, name='LibSVM Support Vector Classification', shortname='LibSVM-SVC', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=False),
 'mlp': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.mlp.MLPClassifier'>, name='Multilayer Percepton', shortname='MLP', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'multinomial_nb': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.multinomial_nb.MultinomialNB'>, name='Multinomial Naive Bayes classifier', shortname='MultinomialNB', output_kind='predictions', supported_inputs=['dense', 'sparse', 'signed data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'passive_aggressive': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.passive_aggressive.PassiveAggressive'>, name='Passive Aggressive Classifier', shortname='PassiveAggressive Classifier', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'qda': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.qda.QDA'>, name='Quadratic Discriminant Analysis', shortname='QDA', output_kind='predictions', supported_inputs=['dense', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'random_forest': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.random_forest.RandomForest'>, name='Random Forest Classifier', shortname='RF', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=True),
 'sgd': ClassifierInfo(type=<class 'autosklearn.pipeline.components.classification.sgd.SGD'>, name='Stochastic Gradient Descent Classifier', shortname='SGD Classifier', output_kind='predictions', supported_inputs=['dense', 'sparse', 'unsigned data'], deterministic=True, handles_binary=True, handles_multiclass=True, handles_multilabel=False)}

The same exists for all classifiers, regressors, data_preprocessors and feature_preprocessors

from autosklearn.info import components

# Same as the classifiers above
all_components = components()
classifiers = all_components.classfiers
feature_preprocessors = all_components.feature_preprocessors

---

Issues

• Data preprocessors are their own beast, there's technically only one datapreprocessor component FeatTypeSplit

  • We've never come up with a nice solution to this as the pipeline is not flexible enough to really represent what we want

• There's a lot of duplication existing already, i.e. handles_spares = True and input = (SPARSE, ...) in get_properties().

  • The dataclasses defined could just be used directly in each component. It makes it type safe, use . notation instead of ["this"].
  • Has the added benefit of really simplifying the stuff in autosklearn.info too.
  • Also means one source of truth for what's allowed in those properties

• I would prefer to do what's below long term, it's a bit nicer to look at. The problem now is that if a user adds a custom component, I would like it to show up when they do components. This is fixable, just noting that it could be changed.

from autosklearn.info import components


# Current requires calling
for name, info in components().classfiers.items():
    # do stuff

# Note how components doesn't need to be called
for name, info in components.classfiers.items():
    # do stuff

---

• Closes Can Autosklearn handle Multi-Class/Multi-Label Classification and which classifiers will it use? #1429

[codecov]

Codecov Report

Merging #1530 (c0061e0) into development (0f1f38a) will decrease coverage by 0.03%.
The diff coverage is 63.27%.

@@               Coverage Diff               @@
##           development    #1530      +/-   ##
===============================================
- Coverage        83.79%   83.75%   -0.04%     
===============================================
  Files              152      154       +2     
  Lines            11667    11730      +63     
  Branches          2037     2049      +12     
===============================================
+ Hits              9776     9825      +49     
- Misses            1343     1359      +16     
+ Partials           548      546       -2     

[auvipy]

It is a huge undertake in one go! it would be better to split the changes in smaller reviewable chunks

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 80.32544% with 266 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.75%. Comparing base (cd4fef9) to head (c0061e0).
⚠️ Report is 176 commits behind head on development.

❌ Your patch status has failed because the patch coverage (80.32%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project status has failed because the head coverage (83.75%) is below the target coverage (84.00%). You can increase the head coverage or adjust the target coverage.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@               Coverage Diff               @@
##           development    #1530      +/-   ##
===============================================
- Coverage        88.13%   83.75%   -4.38%     
===============================================
  Files              140      154      +14     
  Lines            11163    11730     +567     
  Branches             0     2049    +2049     
===============================================
- Hits              9839     9825      -14     
- Misses            1324     1359      +35     
- Partials             0      546     +546     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.