.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples_built/plot_held_out_lasso.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_built_plot_held_out_lasso.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_built_plot_held_out_lasso.py:


Held-out Lasso with HOAG
========================

The canonical sparho example: tune the Lasso regularization strength ``α``
to minimize the mean-squared error on a fixed held-out set, using one
gradient-based outer search instead of a grid sweep.

This is what ``hoag_search`` was written for. Sklearn's ``LassoCV`` is the
grid-search baseline; we compare against it at the end.

.. GENERATED FROM PYTHON SOURCE LINES 12-22

.. code-block:: Python


    import matplotlib.pyplot as plt
    import numpy as np
    from sklearn.datasets import make_regression
    from sklearn.linear_model import LassoCV
    from sklearn.metrics import mean_squared_error

    from sparho import HeldOutMSE, L1, Problem, SquaredLoss, hoag_search
    from sparho.adapters import SklearnLasso








.. GENERATED FROM PYTHON SOURCE LINES 23-24

Synthetic data — 300 samples × 100 features, 10 informative, mild noise.

.. GENERATED FROM PYTHON SOURCE LINES 24-33

.. code-block:: Python

    X, y = make_regression(
        n_samples=300, n_features=100, n_informative=10,
        noise=1.0, random_state=0,
    )

    rng = np.random.default_rng(0)
    perm = rng.permutation(X.shape[0]).astype(np.int32)
    idx_train, idx_val = perm[:200], perm[200:]








.. GENERATED FROM PYTHON SOURCE LINES 34-35

Run sparho — one HOAG outer search in ``log α`` space, starting at α = 1e-2.

.. GENERATED FROM PYTHON SOURCE LINES 35-48

.. code-block:: Python

    problem = Problem(SquaredLoss(), L1(), X, y)
    result = hoag_search(
        problem,
        hp0=1e-2,
        solver=SklearnLasso(tol=1e-8),
        criterion=HeldOutMSE(idx_train, idx_val),
        n_iter=30,
    )

    best_alpha = float(result.best_hyperparam)
    best_mse = mean_squared_error(y[idx_val], X[idx_val] @ result.best_coef)
    print(f"sparho: α = {best_alpha:.4g}   held-out MSE = {best_mse:.4f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    sparho: α = 0.07518   held-out MSE = 0.8320




.. GENERATED FROM PYTHON SOURCE LINES 49-50

Baseline: ``LassoCV`` on a 30-point log grid using the same fold.

.. GENERATED FROM PYTHON SOURCE LINES 50-59

.. code-block:: Python

    alphas_grid = np.logspace(-4, 1, 30)
    cv = LassoCV(
        alphas=alphas_grid, cv=[(idx_train, idx_val)],
        fit_intercept=False, tol=1e-8, max_iter=10_000,
    )
    cv.fit(X, y)
    grid_mse = mean_squared_error(y[idx_val], X[idx_val] @ cv.coef_)
    print(f"LassoCV: α = {cv.alpha_:.4g}   held-out MSE = {grid_mse:.4f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    LassoCV: α = 0.08532   held-out MSE = 0.8632




.. GENERATED FROM PYTHON SOURCE LINES 60-61

Plot the sparho trajectory against the grid baseline.

.. GENERATED FROM PYTHON SOURCE LINES 61-76

.. code-block:: Python

    fig, ax = plt.subplots(figsize=(6, 4))

    xs = [float(r.hyperparam) for r in result.history]
    ys = [r.value for r in result.history]
    ax.plot(xs, ys, "o-", color="C0", label="sparho HOAG trajectory")
    ax.axvline(best_alpha, color="C0", linestyle="--", alpha=0.5)
    ax.axvline(cv.alpha_, color="C1", linestyle="--", alpha=0.5,
               label=f"LassoCV α = {cv.alpha_:.2g}")
    ax.set_xscale("log")
    ax.set_xlabel(r"$\alpha$")
    ax.set_ylabel("held-out MSE")
    ax.set_title("Held-out Lasso: gradient-based search vs. grid")
    ax.legend()
    fig.tight_layout()
    plt.show()



.. image-sg:: /examples_built/images/sphx_glr_plot_held_out_lasso_001.png
   :alt: Held-out Lasso: gradient-based search vs. grid
   :srcset: /examples_built/images/sphx_glr_plot_held_out_lasso_001.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.144 seconds)


.. _sphx_glr_download_examples_built_plot_held_out_lasso.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_held_out_lasso.ipynb <plot_held_out_lasso.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_held_out_lasso.py <plot_held_out_lasso.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_held_out_lasso.zip <plot_held_out_lasso.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_