go home Home | Main Page | Modules | Namespace List | Class Hierarchy | Alphabetical List | Data Structures | File List | Namespace Members | Data Fields | Globals | Related Pages
Public Types | Public Member Functions | Static Public Member Functions | Protected Member Functions | Protected Attributes | Private Attributes
itk::PreconditionedASGDOptimizer Class Reference

#include <itkPreconditionedASGDOptimizer.h>

Detailed Description

This class implements a gradient descent optimizer with adaptive gain.

If $C(x)$ is a cost function that has to be minimized, the following iterative algorithm is used to find the optimal parameters $x$:

\[ x(k+1) = x(k) - a(t_k) dC/dx \]

The gain $a(t_k)$ at each iteration $k$ is defined by:

\[ a(t_k) =  a / (A + t_k + 1)^alpha \]

.

And the time $t_k$ is updated according to:

\[ t_{k+1} = [ t_k + sigmoid( -g_k^T g_{k-1} ) ]^+ \]

where $g_k$ equals $dC/dx$ at iteration $k$. For $t_0$ the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] P. Cruz, Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case. Technical Report, 2005. http://hdl.handle.net/2052/74

[2] S. Klein, J.P.W. Pluim, M. Staring, M.A. Viergever, Adaptive stochastic gradient descent optimisation for image registration International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009. http://dx.doi.org/10.1007/s11263-008-0168-y

It is very suitable to be used in combination with a stochastic estimate of the gradient $dC/dx$. For example, in image registration problems it is often advantageous to compute the metric derivative ( $dC/dx$) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also
VoxelWiseASGD, StandardGradientDescentOptimizer

Definition at line 70 of file itkPreconditionedASGDOptimizer.h.

Inheritance diagram for itk::PreconditionedASGDOptimizer:
Inheritance graph
[legend]

Public Types

using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using Self = PreconditionedASGDOptimizer
 
using Superclass = StandardGradientDescentOptimizer
 
- Public Types inherited from itk::StandardGradientDescentOptimizer
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer
 
using ScaledCostFunctionType = ScaledSingleValuedCostFunction
 
using ScalesType = NonLinearOptimizer::ScalesType
 
using Self = StandardGradientDescentOptimizer
 
enum  StopConditionType
 
using Superclass = GradientDescentOptimizer2
 
- Public Types inherited from itk::GradientDescentOptimizer2
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer
 
using ScaledCostFunctionType = ScaledSingleValuedCostFunction
 
using ScalesType = NonLinearOptimizer::ScalesType
 
using Self = GradientDescentOptimizer2
 
enum  StopConditionType { MaximumNumberOfIterations , MetricError , MinimumStepSize }
 
using Superclass = ScaledSingleValuedNonLinearOptimizer
 
- Public Types inherited from itk::ScaledSingleValuedNonLinearOptimizer
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer
 
using ScaledCostFunctionType = ScaledSingleValuedCostFunction
 
using ScalesType = NonLinearOptimizer::ScalesType
 
using Self = ScaledSingleValuedNonLinearOptimizer
 
using Superclass = SingleValuedNonLinearOptimizer
 

Public Member Functions

virtual const char * GetClassName () const
 
virtual const ParametersType & GetPreconditionVector ()
 
virtual double GetSigmoidMax () const
 
virtual double GetSigmoidMin () const
 
virtual double GetSigmoidScale () const
 
virtual bool GetUseAdaptiveStepSizes () const
 
 ITK_DISALLOW_COPY_AND_MOVE (PreconditionedASGDOptimizer)
 
virtual void SetSigmoidMax (double _arg)
 
virtual void SetSigmoidMin (double _arg)
 
virtual void SetSigmoidScale (double _arg)
 
virtual void SetUseAdaptiveStepSizes (bool _arg)
 
- Public Member Functions inherited from itk::StandardGradientDescentOptimizer
void AdvanceOneStep () override
 
virtual const char * GetClassName () const
 
virtual double GetCurrentTime () const
 
virtual double GetInitialTime () const
 
virtual double GetParam_a () const
 
virtual double GetParam_A () const
 
virtual double GetParam_alpha () const
 
 ITK_DISALLOW_COPY_AND_MOVE (StandardGradientDescentOptimizer)
 
virtual void ResetCurrentTimeToInitialTime ()
 
virtual void SetInitialTime (double _arg)
 
virtual void SetParam_a (double _arg)
 
virtual void SetParam_A (double _arg)
 
virtual void SetParam_alpha (double _arg)
 
void StartOptimization () override
 
- Public Member Functions inherited from itk::GradientDescentOptimizer2
virtual void AdvanceOneStep ()
 
virtual const char * GetClassName () const
 
virtual unsigned int GetCurrentIteration () const
 
virtual const DerivativeType & GetGradient ()
 
virtual const doubleGetLearningRate ()
 
virtual const unsigned long & GetNumberOfIterations ()
 
virtual const DerivativeType & GetSearchDirection ()
 
virtual const StopConditionTypeGetStopCondition ()
 
virtual const doubleGetValue ()
 
 ITK_DISALLOW_COPY_AND_MOVE (GradientDescentOptimizer2)
 
virtual void MetricErrorResponse (ExceptionObject &err)
 
virtual void ResumeOptimization ()
 
virtual void SetLearningRate (double _arg)
 
virtual void SetNumberOfIterations (unsigned long _arg)
 
virtual void SetUseOpenMP (bool _arg)
 
void StartOptimization () override
 
virtual void StopOptimization ()
 
- Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual const char * GetClassName () const
 
const ParametersType & GetCurrentPosition () const override
 
virtual bool GetMaximize () const
 
virtual const ScaledCostFunctionTypeGetScaledCostFunction ()
 
virtual const ParametersType & GetScaledCurrentPosition ()
 
bool GetUseScales () const
 
virtual void InitializeScales ()
 
 ITK_DISALLOW_COPY_AND_MOVE (ScaledSingleValuedNonLinearOptimizer)
 
virtual void MaximizeOff ()
 
virtual void MaximizeOn ()
 
void SetCostFunction (CostFunctionType *costFunction) override
 
virtual void SetMaximize (bool _arg)
 
virtual void SetUseScales (bool arg)
 

Static Public Member Functions

static Pointer New ()
 
- Static Public Member Functions inherited from itk::StandardGradientDescentOptimizer
static Pointer New ()
 
- Static Public Member Functions inherited from itk::GradientDescentOptimizer2
static Pointer New ()
 
- Static Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
static Pointer New ()
 

Protected Member Functions

 PreconditionedASGDOptimizer ()
 
void UpdateCurrentTime () override
 
 ~PreconditionedASGDOptimizer () override=default
 
- Protected Member Functions inherited from itk::StandardGradientDescentOptimizer
virtual double Compute_a (double k) const
 
 StandardGradientDescentOptimizer ()
 
virtual void UpdateCurrentTime ()
 
 ~StandardGradientDescentOptimizer () override=default
 
- Protected Member Functions inherited from itk::GradientDescentOptimizer2
 GradientDescentOptimizer2 ()
 
void PrintSelf (std::ostream &os, Indent indent) const override
 
 ~GradientDescentOptimizer2 () override=default
 
- Protected Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual void GetScaledDerivative (const ParametersType &parameters, DerivativeType &derivative) const
 
virtual MeasureType GetScaledValue (const ParametersType &parameters) const
 
virtual void GetScaledValueAndDerivative (const ParametersType &parameters, MeasureType &value, DerivativeType &derivative) const
 
void PrintSelf (std::ostream &os, Indent indent) const override
 
 ScaledSingleValuedNonLinearOptimizer ()
 
void SetCurrentPosition (const ParametersType &param) override
 
virtual void SetScaledCurrentPosition (const ParametersType &parameters)
 
 ~ScaledSingleValuedNonLinearOptimizer () override=default
 

Protected Attributes

ParametersType m_PreconditionVector
 
DerivativeType m_PreviousSearchDirection
 
std::string m_StepSizeStrategy
 
- Protected Attributes inherited from itk::StandardGradientDescentOptimizer
double m_CurrentTime { 0.0 }
 
bool m_UseConstantStep { false }
 
- Protected Attributes inherited from itk::GradientDescentOptimizer2
DerivativeType m_Gradient
 
DerivativeType m_SearchDirection
 
StopConditionType m_StopCondition { MaximumNumberOfIterations }
 
- Protected Attributes inherited from itk::ScaledSingleValuedNonLinearOptimizer
ScaledCostFunctionPointer m_ScaledCostFunction
 
ParametersType m_ScaledCurrentPosition
 

Private Attributes

double m_SigmoidMax { 1.0 }
 
double m_SigmoidMin { -0.8 }
 
double m_SigmoidScale { 1e-8 }
 
bool m_UseAdaptiveStepSizes { true }
 

Member Typedef Documentation

◆ ConstPointer

Definition at line 79 of file itkPreconditionedASGDOptimizer.h.

◆ Pointer

Definition at line 78 of file itkPreconditionedASGDOptimizer.h.

◆ Self

Standard ITK.

Definition at line 76 of file itkPreconditionedASGDOptimizer.h.

◆ Superclass

Definition at line 77 of file itkPreconditionedASGDOptimizer.h.

Constructor & Destructor Documentation

◆ PreconditionedASGDOptimizer()

itk::PreconditionedASGDOptimizer::PreconditionedASGDOptimizer ( )
protected

◆ ~PreconditionedASGDOptimizer()

itk::PreconditionedASGDOptimizer::~PreconditionedASGDOptimizer ( )
overrideprotecteddefault

Member Function Documentation

◆ GetClassName()

virtual const char * itk::PreconditionedASGDOptimizer::GetClassName ( ) const
virtual

Run-time type information (and related methods).

Reimplemented from itk::StandardGradientDescentOptimizer.

Reimplemented in elastix::PreconditionedStochasticGradientDescent< TElastix >.

◆ GetPreconditionVector()

virtual const ParametersType & itk::PreconditionedASGDOptimizer::GetPreconditionVector ( )
virtual

Get current gradient.

◆ GetSigmoidMax()

virtual double itk::PreconditionedASGDOptimizer::GetSigmoidMax ( ) const
virtual

◆ GetSigmoidMin()

virtual double itk::PreconditionedASGDOptimizer::GetSigmoidMin ( ) const
virtual

◆ GetSigmoidScale()

virtual double itk::PreconditionedASGDOptimizer::GetSigmoidScale ( ) const
virtual

◆ GetUseAdaptiveStepSizes()

virtual bool itk::PreconditionedASGDOptimizer::GetUseAdaptiveStepSizes ( ) const
virtual

◆ ITK_DISALLOW_COPY_AND_MOVE()

itk::PreconditionedASGDOptimizer::ITK_DISALLOW_COPY_AND_MOVE ( PreconditionedASGDOptimizer  )

◆ New()

static Pointer itk::PreconditionedASGDOptimizer::New ( )
static

Method for creation through the object factory.

◆ SetSigmoidMax()

virtual void itk::PreconditionedASGDOptimizer::SetSigmoidMax ( double  _arg)
virtual

Set/Get the maximum of the sigmoid. Should be >0. Default: 1.0

◆ SetSigmoidMin()

virtual void itk::PreconditionedASGDOptimizer::SetSigmoidMin ( double  _arg)
virtual

Set/Get the maximum of the sigmoid. Should be <0. Default: -0.8

◆ SetSigmoidScale()

virtual void itk::PreconditionedASGDOptimizer::SetSigmoidScale ( double  _arg)
virtual

Set/Get the scaling of the sigmoid width. Large values cause a more wide sigmoid. Default: 1e-8. Should be >0.

◆ SetUseAdaptiveStepSizes()

virtual void itk::PreconditionedASGDOptimizer::SetUseAdaptiveStepSizes ( bool  _arg)
virtual

Set/Get whether the adaptive step size mechanism is desired. Default: true

◆ UpdateCurrentTime()

void itk::PreconditionedASGDOptimizer::UpdateCurrentTime ( )
overrideprotectedvirtual

Function to update the current time If UseAdaptiveStepSizes is false this function just increments the CurrentTime by $E_0 = (sigmoid_{max} + sigmoid_{min})/2$. Else, the CurrentTime is updated according to:
time = max[ 0, time + sigmoid( -gradient*previousgradient) ]
In that case, also the m_PreviousGradient is updated.

Reimplemented from itk::StandardGradientDescentOptimizer.

Field Documentation

◆ m_PreconditionVector

ParametersType itk::PreconditionedASGDOptimizer::m_PreconditionVector
protected

Definition at line 135 of file itkPreconditionedASGDOptimizer.h.

◆ m_PreviousSearchDirection

DerivativeType itk::PreconditionedASGDOptimizer::m_PreviousSearchDirection
protected

The PreviousGradient, necessary for the CruzAcceleration

Definition at line 134 of file itkPreconditionedASGDOptimizer.h.

◆ m_SigmoidMax

double itk::PreconditionedASGDOptimizer::m_SigmoidMax { 1.0 }
private

Definition at line 141 of file itkPreconditionedASGDOptimizer.h.

◆ m_SigmoidMin

double itk::PreconditionedASGDOptimizer::m_SigmoidMin { -0.8 }
private

Definition at line 142 of file itkPreconditionedASGDOptimizer.h.

◆ m_SigmoidScale

double itk::PreconditionedASGDOptimizer::m_SigmoidScale { 1e-8 }
private

Definition at line 143 of file itkPreconditionedASGDOptimizer.h.

◆ m_StepSizeStrategy

std::string itk::PreconditionedASGDOptimizer::m_StepSizeStrategy
protected

Definition at line 136 of file itkPreconditionedASGDOptimizer.h.

◆ m_UseAdaptiveStepSizes

bool itk::PreconditionedASGDOptimizer::m_UseAdaptiveStepSizes { true }
private

Settings

Definition at line 140 of file itkPreconditionedASGDOptimizer.h.



Generated on 2023-01-13 for elastix by doxygen 1.9.6 elastix logo