13.5 Combating non-convexity with regularization

Press the button 'Toggle code' below to toggle code on and off for entire this presentation.

from IPython.display import display
from IPython.display import HTML
import IPython.core.display as di # Example: di.display_html('<h3>%s:</h3>' % str, raw=True)

# This line will hide code by default when the notebook is eåxported as HTML
di.display_html('<script>jQuery(function() {if (jQuery("body.notebook_app").length == 0) { jQuery(".input_area").toggle(); jQuery(".prompt").toggle();}});</script>', raw=True)

# This line will add a button to toggle visibility of code blocks, for use with the HTML export version
di.display_html('''<button onclick="jQuery('.input_area').toggle(); jQuery('.prompt').toggle();">Toggle code</button>''', raw=True)

Toggle code

Example 1. The effect of regularization on flat regions of a non-convex function

Moving the slider widget from left to right we gradually increase $\lambda$ from $0$ to $1$.

# what function should we play with?  Defined in the next line.
g1 = lambda w: -np.maximum(0,.5*np.exp(-w)*np.sin(4*np.pi*(w - 0.1)))**2
g2 = lambda w: w**2

# create an instance of the visualizer with this function
test = basics_library.convex_function_addition_2d.visualizer()

# plot away
test.draw_it(g1 = g1,g2 = g2,num_slides = 50,min_range = -0.5,max_range = 0.5, alpha_range=(0,1), mode='regularization', title3='$g_1 + \\lambda\,g_2$')

Example 2. The effect of regularization on saddle points of a non-convex function

Moving the slider widget from left to right we gradually increase $\lambda$ from $0$ to $1$.

# what function should we play with?  Defined in the next line.
g1 = lambda w: np.maximum(0,(3*w - 2.3)**3 + 1)**2 + np.maximum(0, (-3*w + 0.7)**3 + 1)**2
g2 = lambda w: w**2

# create an instance of the visualizer with this function
test = basics_library.convex_function_addition_2d.visualizer()

# plot away
test.draw_it(g1 = g1,g2 = g2,num_slides = 50,min_range = 0,max_range = 1, alpha_range=(0,1), mode='regularization', title3='$g_1 + \\lambda\,g_2$')

Example 3. Bacterial growth

test = basics_library.regularization.visualizer()

w_init_1 = [0,2]
w_init_2 = [0,-2]
lam_range = 0
max_its = 5000

test.run_example(w_init_1 = w_init_1, w_init_2 = w_init_2, max_its = max_its)