.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples_built/plot_sparse_logreg.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_built_plot_sparse_logreg.py: Sparse logistic regression ========================== Held-out logistic loss on a binary classification problem with an L1 penalty. The Problem is ``(LogisticLoss, L1, X, y)`` with ``y ∈ {−1, +1}``; the rest of the bilevel machinery is unchanged. .. GENERATED FROM PYTHON SOURCE LINES 9-16 .. code-block:: Python import matplotlib.pyplot as plt import numpy as np from sklearn.datasets import make_classification from sparho import L1, HeldOutLogistic, LogisticLoss, Problem, hoag_search from sparho.adapters import SklearnLogisticRegression .. GENERATED FROM PYTHON SOURCE LINES 17-18 Binary classification: 250 samples × 50 features, 8 informative. .. GENERATED FROM PYTHON SOURCE LINES 18-32 .. code-block:: Python X, y01 = make_classification( n_samples=250, n_features=50, n_informative=8, n_redundant=5, random_state=0, ) # sparho's LogisticLoss expects ±1 labels. y = (2 * y01 - 1).astype(np.float64) rng = np.random.default_rng(0) perm = rng.permutation(X.shape[0]).astype(np.int32) idx_train, idx_val = perm[:180], perm[180:] .. GENERATED FROM PYTHON SOURCE LINES 33-36 Outer search. ``SklearnLogisticRegression`` uses liblinear under the hood; liblinear doesn't support warm-start, so ``x0`` is ignored — only ``tol`` is meaningful here. .. GENERATED FROM PYTHON SOURCE LINES 36-49 .. code-block:: Python problem = Problem(LogisticLoss(), L1(), X, y) result = hoag_search( problem, hp0=1e-1, solver=SklearnLogisticRegression(tol=1e-8), criterion=HeldOutLogistic(idx_train, idx_val), n_iter=20, ) print(f"sparho: α = {float(result.best_hyperparam):.4g}") print(f" held-out loss = {result.history[-1].value:.4f}") print(f" |active set| = {np.count_nonzero(result.best_coef)}") .. rst-class:: sphx-glr-script-out .. code-block:: none /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None. warnings.warn( /home/docs/checkouts/readthedocs.org/user_builds/sparho/envs/latest/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only. warnings.warn( sparho: α = 2.88 held-out loss = 0.3961 |active set| = 23 .. GENERATED FROM PYTHON SOURCE LINES 50-51 Plot the outer-loop trajectory. .. GENERATED FROM PYTHON SOURCE LINES 51-66 .. code-block:: Python fig, (ax_a, ax_b) = plt.subplots(1, 2, figsize=(10, 4)) xs = [float(r.hyperparam) for r in result.history] ys = [r.value for r in result.history] ax_a.semilogx(xs, ys, "o-") ax_a.set_xlabel(r"$\alpha$") ax_a.set_ylabel("held-out logistic loss") ax_a.set_title("Outer-search trajectory") ax_b.stem(np.arange(X.shape[1]), result.best_coef, basefmt=" ") ax_b.set_xlabel("feature index") ax_b.set_ylabel(r"$\beta_j^\star$") ax_b.set_title("Recovered coefficients") fig.tight_layout() plt.show() .. image-sg:: /examples_built/images/sphx_glr_plot_sparse_logreg_001.png :alt: Outer-search trajectory, Recovered coefficients :srcset: /examples_built/images/sphx_glr_plot_sparse_logreg_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.658 seconds) .. _sphx_glr_download_examples_built_plot_sparse_logreg.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_sparse_logreg.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_sparse_logreg.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_sparse_logreg.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_