So far, we've seen how to model some (time-series) data with a feature-embedded and feature-weighted regression model using puffins. We've seen how different types of regression models perform on the same data, and we've also see how to tune the hyper-parameters of our model.
Now, in this notebook, we're going to look at three different methods for assessing the uncertainties and standard errors on the terms in our models.
In [1]:
%matplotlib widget
import numpy as np
import matplotlib.pyplot as plt
from puffins.tuner import Tuner
from puffins.weight_functions import matern32
from puffins.data import TimeSeries,LinearModel
from puffins.uncertainties import UncertaintyEstimator
from puffins.basis_functions import basis_constant, basis_linear
np.random.seed(42) # to enforce reproducibility
RCOND = 1e-14 # numerical rounding for matrix conditioning
plotnum=1
Instantiate the data and model classes¶
In [2]:
time, flux = np.loadtxt('../data/ugru.dat').T
flux_err = np.random.normal(0,0.005,len(flux))
period = 1.88045
n_harmonics = 100
feature_weighting_width = 0.5
Parameter tuning¶
As we saw in the last notebook, the difference between randomly chosen and optimised parameters can be very stark on the resulting model. Therefore, we want to make sure that our model is as good a representation of the data as possible before we start to estimate the model uncertaintes.
In [3]:
data = TimeSeries(time, flux)
fwls = LinearModel('fw', basis_functions=[basis_constant,], feature_embedding='fourier',
feature_weighting_function=matern32, feature_weighting_width=feature_weighting_width,
period=period, n_harmonics=100, W=None)
In [4]:
joint_ = {'feature_weighting_width': [0.001,1,'uniform'], 'period': [1.87,1.89,'uniform']}
joint_tuner = Tuner(fwls, hyperpars=joint_, n_trials=200, direction='minimize')
joint_tuner.run_tune(data.predictors, data.targets)
[I 2025-02-07 15:51:49,591] A new study created in memory with name: no-name-53881c9e-3212-4602-a84b-eb101cdb5623
[I 2025-02-07 15:51:49,683] Trial 0 finished with value: 0.007463692239346467 and parameters: {'feature_weighting_width': 0.7142292680133613, 'period': 1.8739029489452292}. Best is trial 0 with value: 0.007463692239346467.
[I 2025-02-07 15:51:49,754] Trial 1 finished with value: 0.00894381533422166 and parameters: {'feature_weighting_width': 0.8155915998544844, 'period': 1.887015697783962}. Best is trial 0 with value: 0.007463692239346467.
[I 2025-02-07 15:51:49,895] Trial 2 finished with value: 0.0018672495893446296 and parameters: {'feature_weighting_width': 0.22535016872150446, 'period': 1.8753074657469764}. Best is trial 2 with value: 0.0018672495893446296.
[I 2025-02-07 15:51:49,972] Trial 3 finished with value: 0.0017562473626849492 and parameters: {'feature_weighting_width': 0.3622549112180056, 'period': 1.8763724322012085}. Best is trial 3 with value: 0.0017562473626849492.
[I 2025-02-07 15:51:50,038] Trial 4 finished with value: 0.008907579631977346 and parameters: {'feature_weighting_width': 0.7745854642976996, 'period': 1.8883942842481496}. Best is trial 3 with value: 0.0017562473626849492.
[I 2025-02-07 15:51:50,096] Trial 5 finished with value: 0.0016033932517117598 and parameters: {'feature_weighting_width': 0.12467915556778421, 'period': 1.8852900693608576}. Best is trial 5 with value: 0.0016033932517117598.
[I 2025-02-07 15:51:50,160] Trial 6 finished with value: 0.009428374424861115 and parameters: {'feature_weighting_width': 0.9174945615092488, 'period': 1.8810724759614799}. Best is trial 5 with value: 0.0016033932517117598.
[I 2025-02-07 15:51:50,342] Trial 7 finished with value: 0.00531198254158268 and parameters: {'feature_weighting_width': 0.06320000990716065, 'period': 1.8710873115984938}. Best is trial 5 with value: 0.0016033932517117598.
[I 2025-02-07 15:51:50,415] Trial 8 finished with value: 0.0005788553248969712 and parameters: {'feature_weighting_width': 0.2725046967273033, 'period': 1.8828545917896988}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,495] Trial 9 finished with value: 0.0053531717293669396 and parameters: {'feature_weighting_width': 0.6323128044657571, 'period': 1.8759320203827534}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,575] Trial 10 finished with value: 0.001113971249808822 and parameters: {'feature_weighting_width': 0.3842842258442287, 'period': 1.882356230290984}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,654] Trial 11 finished with value: 0.001889854507876081 and parameters: {'feature_weighting_width': 0.455021129373569, 'period': 1.8823414200400896}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,794] Trial 12 finished with value: 0.0007900023856134947 and parameters: {'feature_weighting_width': 0.26763464214982147, 'period': 1.8834747147623814}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,865] Trial 13 finished with value: 0.001120325739881165 and parameters: {'feature_weighting_width': 0.25985465459593865, 'period': 1.8842509934741465}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:50,940] Trial 14 finished with value: 0.003096206294383426 and parameters: {'feature_weighting_width': 0.542791184373463, 'period': 1.8789457157181675}. Best is trial 8 with value: 0.0005788553248969712.
[I 2025-02-07 15:51:51,060] Trial 15 finished with value: 0.00021957031941328978 and parameters: {'feature_weighting_width': 0.23733724262004913, 'period': 1.8791546528924448}. Best is trial 15 with value: 0.00021957031941328978.
[I 2025-02-07 15:51:51,136] Trial 16 finished with value: 0.00015650400034216634 and parameters: {'feature_weighting_width': 0.015227684341822645, 'period': 1.8790308848809982}. Best is trial 16 with value: 0.00015650400034216634.
[I 2025-02-07 15:51:51,188] Trial 17 finished with value: 0.00017888639946663976 and parameters: {'feature_weighting_width': 0.021183047105680163, 'period': 1.8789266013399788}. Best is trial 16 with value: 0.00015650400034216634.
[I 2025-02-07 15:51:51,241] Trial 18 finished with value: 0.0004194684944008344 and parameters: {'feature_weighting_width': 0.05331605751328633, 'period': 1.8780706394472584}. Best is trial 16 with value: 0.00015650400034216634.
[I 2025-02-07 15:51:51,311] Trial 19 finished with value: 0.0038696866657983242 and parameters: {'feature_weighting_width': 0.006250874717483172, 'period': 1.8726732808596438}. Best is trial 16 with value: 0.00015650400034216634.
[I 2025-02-07 15:51:51,473] Trial 20 finished with value: 0.000937832983108492 and parameters: {'feature_weighting_width': 0.1414368292828861, 'period': 1.8768414514477105}. Best is trial 16 with value: 0.00015650400034216634.
[I 2025-02-07 15:51:51,559] Trial 21 finished with value: 2.5277787570213525e-05 and parameters: {'feature_weighting_width': 0.15136189015700036, 'period': 1.88012800167119}. Best is trial 21 with value: 2.5277787570213525e-05.
[I 2025-02-07 15:51:51,623] Trial 22 finished with value: 1.1320867511386637e-05 and parameters: {'feature_weighting_width': 0.1250848767290107, 'period': 1.8805482834926208}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:51,743] Trial 23 finished with value: 6.448207320959737e-05 and parameters: {'feature_weighting_width': 0.13189223675861475, 'period': 1.8813202188953035}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:51,864] Trial 24 finished with value: 8.993742418539841e-05 and parameters: {'feature_weighting_width': 0.1672303038430761, 'period': 1.881432692244943}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:51,927] Trial 25 finished with value: 0.0006017381926286397 and parameters: {'feature_weighting_width': 0.349853611892888, 'period': 1.8806113184398576}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,088] Trial 26 finished with value: 0.0018438846712448566 and parameters: {'feature_weighting_width': 0.11978601777579967, 'period': 1.8856634366308669}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,195] Trial 27 finished with value: 0.0029705765108141574 and parameters: {'feature_weighting_width': 0.5139474267316039, 'period': 1.8775643863868405}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,248] Trial 28 finished with value: 3.1407129378119376e-05 and parameters: {'feature_weighting_width': 0.1790754947073498, 'period': 1.8803058054242805}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,310] Trial 29 finished with value: 0.003991498837324977 and parameters: {'feature_weighting_width': 0.4379746579133958, 'period': 1.8737040040771717}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,366] Trial 30 finished with value: 6.201104381151648e-05 and parameters: {'feature_weighting_width': 0.2009233443158651, 'period': 1.8800031935130914}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,425] Trial 31 finished with value: 3.848470084135628e-05 and parameters: {'feature_weighting_width': 0.1908952575384773, 'period': 1.8805718700652039}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,490] Trial 32 finished with value: 0.001301844018936123 and parameters: {'feature_weighting_width': 0.318048052446228, 'period': 1.8841983799029804}. Best is trial 22 with value: 1.1320867511386637e-05.
[I 2025-02-07 15:51:52,547] Trial 33 finished with value: 8.781550750759097e-06 and parameters: {'feature_weighting_width': 0.087034054512731, 'period': 1.8803458860831874}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,629] Trial 34 finished with value: 0.002299124712351358 and parameters: {'feature_weighting_width': 0.10157791284873216, 'period': 1.874629046226037}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,692] Trial 35 finished with value: 0.0006668470171134101 and parameters: {'feature_weighting_width': 0.0753190265815248, 'period': 1.8774227866622604}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,761] Trial 36 finished with value: 0.0022132910432548203 and parameters: {'feature_weighting_width': 0.309656025038319, 'period': 1.8858708364066297}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,823] Trial 37 finished with value: 0.0035654744568580196 and parameters: {'feature_weighting_width': 0.1813625243653378, 'period': 1.887919634820463}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,879] Trial 38 finished with value: 0.010797643336738362 and parameters: {'feature_weighting_width': 0.9955915751015048, 'period': 1.8780632247787106}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:52,946] Trial 39 finished with value: 4.669525015124249e-05 and parameters: {'feature_weighting_width': 0.08943006787456492, 'period': 1.8797359671419827}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,010] Trial 40 finished with value: 0.005658883022774872 and parameters: {'feature_weighting_width': 0.6904484262482715, 'period': 1.8824800857261892}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,078] Trial 41 finished with value: 5.249703419080934e-05 and parameters: {'feature_weighting_width': 0.20585222811419973, 'period': 1.8805865211977104}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,144] Trial 42 finished with value: 9.011184637828036e-05 and parameters: {'feature_weighting_width': 0.17210194852584176, 'period': 1.8814168244183218}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,225] Trial 43 finished with value: 1.6555771955640446e-05 and parameters: {'feature_weighting_width': 0.06238994003747805, 'period': 1.88010685807171}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,303] Trial 44 finished with value: 0.005089009424593475 and parameters: {'feature_weighting_width': 0.0464919461413176, 'period': 1.8896359017005768}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,357] Trial 45 finished with value: 0.0006389063981288847 and parameters: {'feature_weighting_width': 0.07160012610798462, 'period': 1.8834508416056746}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,420] Trial 46 finished with value: 0.0012146773616194333 and parameters: {'feature_weighting_width': 0.13423192651415156, 'period': 1.8763128368278006}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,478] Trial 47 finished with value: 0.0003048589543317039 and parameters: {'feature_weighting_width': 0.24917506722514113, 'period': 1.8820834787793144}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,534] Trial 48 finished with value: 0.003709892777415192 and parameters: {'feature_weighting_width': 0.587075125243608, 'period': 1.8801970741556822}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,597] Trial 49 finished with value: 0.0015803917337608435 and parameters: {'feature_weighting_width': 0.42178260121613176, 'period': 1.8782609863148139}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,663] Trial 50 finished with value: 0.00016913910315009085 and parameters: {'feature_weighting_width': 0.22760994032483256, 'period': 1.879350304530087}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,725] Trial 51 finished with value: 0.0002996821077889422 and parameters: {'feature_weighting_width': 0.29825714771772055, 'period': 1.8808478155687764}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,781] Trial 52 finished with value: 0.00013318874613763506 and parameters: {'feature_weighting_width': 0.15461925829418616, 'period': 1.8817342015580982}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,837] Trial 53 finished with value: 0.00817636310024567 and parameters: {'feature_weighting_width': 0.8421632404623722, 'period': 1.8800863461291466}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,894] Trial 54 finished with value: 0.00044698347736040834 and parameters: {'feature_weighting_width': 0.030315525533423864, 'period': 1.8829483640871074}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:53,942] Trial 55 finished with value: 0.00018015468288674362 and parameters: {'feature_weighting_width': 0.08647925876250903, 'period': 1.878923435227183}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:54,006] Trial 56 finished with value: 0.0011064953802915947 and parameters: {'feature_weighting_width': 0.20666398796768826, 'period': 1.8843792768656296}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:54,055] Trial 57 finished with value: 0.0008383204086685374 and parameters: {'feature_weighting_width': 0.11204987510556452, 'period': 1.8770416082349006}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:54,113] Trial 58 finished with value: 0.0008750073836745291 and parameters: {'feature_weighting_width': 0.35607028513067684, 'period': 1.8786050920698794}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:54,172] Trial 59 finished with value: 0.0002516784428900503 and parameters: {'feature_weighting_width': 0.27524938214479855, 'period': 1.8795904534114882}. Best is trial 33 with value: 8.781550750759097e-06.
[I 2025-02-07 15:51:54,221] Trial 60 finished with value: 8.018765631093213e-06 and parameters: {'feature_weighting_width': 0.03683699634785163, 'period': 1.880605441606501}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,279] Trial 61 finished with value: 2.334440750697982e-05 and parameters: {'feature_weighting_width': 0.0260792497907556, 'period': 1.880947073983876}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,340] Trial 62 finished with value: 0.00014462292955258149 and parameters: {'feature_weighting_width': 0.010192847084188841, 'period': 1.8818473483139624}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,399] Trial 63 finished with value: 2.3776195161156367e-05 and parameters: {'feature_weighting_width': 0.0027394382954903304, 'period': 1.8809533797390483}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,472] Trial 64 finished with value: 4.123411110458764e-05 and parameters: {'feature_weighting_width': 0.05749626643506177, 'period': 1.8811540659443853}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,542] Trial 65 finished with value: 0.0002759838764894161 and parameters: {'feature_weighting_width': 0.03555509403161836, 'period': 1.8824009813494513}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,620] Trial 66 finished with value: 0.0005953432758181449 and parameters: {'feature_weighting_width': 0.011674910307057057, 'period': 1.8833442021935605}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,685] Trial 67 finished with value: 7.265895490804223e-05 and parameters: {'feature_weighting_width': 0.10830519319446277, 'period': 1.8795300462474844}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,750] Trial 68 finished with value: 2.4555498052817427e-05 and parameters: {'feature_weighting_width': 0.04953764492095861, 'period': 1.8809609707150077}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,831] Trial 69 finished with value: 0.0003860432889533246 and parameters: {'feature_weighting_width': 0.004231094669024174, 'period': 1.8827675455631665}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,902] Trial 70 finished with value: 1.6082884719640924e-05 and parameters: {'feature_weighting_width': 0.0491946693823168, 'period': 1.8808241657779243}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:54,962] Trial 71 finished with value: 5.879066964433076e-05 and parameters: {'feature_weighting_width': 0.06099189736149892, 'period': 1.8813117587450103}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:55,031] Trial 72 finished with value: 2.1905020278146995e-05 and parameters: {'feature_weighting_width': 0.04282648015881943, 'period': 1.880924015033114}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:55,098] Trial 73 finished with value: 0.00018011027306237147 and parameters: {'feature_weighting_width': 0.09010300360300214, 'period': 1.882013165351968}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:55,317] Trial 74 finished with value: 0.005553060033462227 and parameters: {'feature_weighting_width': 0.13013861906929214, 'period': 1.870833347279923}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:55,363] Trial 75 finished with value: 0.00029854417106250816 and parameters: {'feature_weighting_width': 0.0014177791540992482, 'period': 1.8784565144278642}. Best is trial 60 with value: 8.018765631093213e-06.
[I 2025-02-07 15:51:55,419] Trial 76 finished with value: 7.516394693130008e-06 and parameters: {'feature_weighting_width': 0.04785417941283544, 'period': 1.8805671951664862}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,481] Trial 77 finished with value: 0.000539685422898226 and parameters: {'feature_weighting_width': 0.03926266322493307, 'period': 1.8777374793561241}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,542] Trial 78 finished with value: 2.9080473387378646e-05 and parameters: {'feature_weighting_width': 0.07400874800110915, 'period': 1.879920873284488}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,692] Trial 79 finished with value: 0.00017191908732147785 and parameters: {'feature_weighting_width': 0.1111449293240216, 'period': 1.8789667626440223}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,834] Trial 80 finished with value: 1.6785613071501408e-05 and parameters: {'feature_weighting_width': 0.15028310451038576, 'period': 1.8805228796351519}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,898] Trial 81 finished with value: 2.0201269803990854e-05 and parameters: {'feature_weighting_width': 0.15101990176431457, 'period': 1.8806852049211091}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:55,972] Trial 82 finished with value: 2.9148826000625977e-05 and parameters: {'feature_weighting_width': 0.1651949830202159, 'period': 1.8801579484650575}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,089] Trial 83 finished with value: 1.2152273487093223e-05 and parameters: {'feature_weighting_width': 0.1306258021543779, 'period': 1.8805397651520455}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,197] Trial 84 finished with value: 0.0001168785217123821 and parameters: {'feature_weighting_width': 0.14695204395315892, 'period': 1.8816544690984771}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,275] Trial 85 finished with value: 8.443586592346852e-06 and parameters: {'feature_weighting_width': 0.09863691732976826, 'period': 1.88041383995098}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,336] Trial 86 finished with value: 7.218555472765528e-05 and parameters: {'feature_weighting_width': 0.07966161123481565, 'period': 1.8795237246257874}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,433] Trial 87 finished with value: 0.00012837492430066511 and parameters: {'feature_weighting_width': 0.12447457581373242, 'period': 1.8791921289709028}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,524] Trial 88 finished with value: 8.504799727413666e-06 and parameters: {'feature_weighting_width': 0.10091258207847503, 'period': 1.880424486159846}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,633] Trial 89 finished with value: 0.0008135117261008375 and parameters: {'feature_weighting_width': 0.0906022691211178, 'period': 1.883848194650478}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,756] Trial 90 finished with value: 0.0003249665903089688 and parameters: {'feature_weighting_width': 0.2216572818674003, 'period': 1.8785737322905938}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:56,981] Trial 91 finished with value: 8.652694599147384e-06 and parameters: {'feature_weighting_width': 0.104511020669131, 'period': 1.8804439327688807}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:57,542] Trial 92 finished with value: 9.830679324218313e-06 and parameters: {'feature_weighting_width': 0.10640492912304177, 'period': 1.880346194233935}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:57,617] Trial 93 finished with value: 0.00010579509025998834 and parameters: {'feature_weighting_width': 0.1084037322934506, 'period': 1.8816256588292288}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:57,817] Trial 94 finished with value: 7.051902570550429e-05 and parameters: {'feature_weighting_width': 0.18343371039580164, 'period': 1.8797388086755062}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:57,886] Trial 95 finished with value: 9.055076270730192e-06 and parameters: {'feature_weighting_width': 0.09665613285841736, 'period': 1.8803546164665277}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:57,953] Trial 96 finished with value: 0.00021149458542818616 and parameters: {'feature_weighting_width': 0.13200684110575533, 'period': 1.8821340174707082}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,015] Trial 97 finished with value: 7.87671869810187e-06 and parameters: {'feature_weighting_width': 0.09145672273300724, 'period': 1.8804608894815305}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,079] Trial 98 finished with value: 0.00040940345426612723 and parameters: {'feature_weighting_width': 0.10241326268172274, 'period': 1.8828358645865557}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,150] Trial 99 finished with value: 0.00011487514420108117 and parameters: {'feature_weighting_width': 0.07075576670447942, 'period': 1.8792497440290385}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,217] Trial 100 finished with value: 0.00011059828532994726 and parameters: {'feature_weighting_width': 0.19770413308172974, 'period': 1.8814385897423036}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,287] Trial 101 finished with value: 1.1560914003334026e-05 and parameters: {'feature_weighting_width': 0.11973721586511718, 'period': 1.8803264331366742}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,349] Trial 102 finished with value: 1.8552568816580383e-05 and parameters: {'feature_weighting_width': 0.09570578232233697, 'period': 1.8800881388613413}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,416] Trial 103 finished with value: 2.5585542825847555e-05 and parameters: {'feature_weighting_width': 0.1715354054831614, 'period': 1.8804008504993794}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,491] Trial 104 finished with value: 4.91900639195321e-05 and parameters: {'feature_weighting_width': 0.07014502267656178, 'period': 1.8797081778347393}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,561] Trial 105 finished with value: 0.005836389769211869 and parameters: {'feature_weighting_width': 0.6938841259124642, 'period': 1.8778740808314955}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,639] Trial 106 finished with value: 0.000238893247478792 and parameters: {'feature_weighting_width': 0.027463245796261374, 'period': 1.878675950812279}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,712] Trial 107 finished with value: 7.038664314205619e-05 and parameters: {'feature_weighting_width': 0.11827575919704122, 'period': 1.881382304136828}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,798] Trial 108 finished with value: 0.0003019486162001511 and parameters: {'feature_weighting_width': 0.08644548667468398, 'period': 1.8824918535180772}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,862] Trial 109 finished with value: 6.804085959400899e-05 and parameters: {'feature_weighting_width': 0.21681470451792229, 'period': 1.8802493841995298}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,922] Trial 110 finished with value: 0.002112954408767079 and parameters: {'feature_weighting_width': 0.4813672109499289, 'period': 1.87941921784325}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:58,985] Trial 111 finished with value: 4.7761198918269164e-05 and parameters: {'feature_weighting_width': 0.12997969986214175, 'period': 1.8811745917460532}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:59,043] Trial 112 finished with value: 7.563254308111391e-06 and parameters: {'feature_weighting_width': 0.05865746137053096, 'period': 1.8803792825153471}. Best is trial 76 with value: 7.516394693130008e-06.
[I 2025-02-07 15:51:59,105] Trial 113 finished with value: 7.275040351762602e-06 and parameters: {'feature_weighting_width': 0.06000087859634762, 'period': 1.880408466190912}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,162] Trial 114 finished with value: 0.0001478710600178204 and parameters: {'feature_weighting_width': 0.028010806884195577, 'period': 1.881863575743941}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,227] Trial 115 finished with value: 3.398330338046797e-05 and parameters: {'feature_weighting_width': 0.05818333401545743, 'period': 1.879858923070644}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,303] Trial 116 finished with value: 0.00018178554768115258 and parameters: {'feature_weighting_width': 0.03793613612207893, 'period': 1.8789137225824932}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,370] Trial 117 finished with value: 0.006869875731998312 and parameters: {'feature_weighting_width': 0.767009934345162, 'period': 1.8806083171964403}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,439] Trial 118 finished with value: 0.004073946242315411 and parameters: {'feature_weighting_width': 0.09428918974915955, 'period': 1.8724404197442392}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,506] Trial 119 finished with value: 3.66419129390512e-05 and parameters: {'feature_weighting_width': 0.0552554082491417, 'period': 1.8811067082527437}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,583] Trial 120 finished with value: 5.0911566010062923e-05 and parameters: {'feature_weighting_width': 0.1621378989429833, 'period': 1.8798228536006467}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,655] Trial 121 finished with value: 8.875587739730634e-06 and parameters: {'feature_weighting_width': 0.06847217330099196, 'period': 1.8803157625964166}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,718] Trial 122 finished with value: 7.605581185886681e-06 and parameters: {'feature_weighting_width': 0.07349042733265858, 'period': 1.8805321523291334}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,780] Trial 123 finished with value: 1.5607425632370893e-05 and parameters: {'feature_weighting_width': 0.0804816646402872, 'period': 1.8808021367243113}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,848] Trial 124 finished with value: 0.00013006741514524597 and parameters: {'feature_weighting_width': 0.06294566032789106, 'period': 1.8791657210710857}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,920] Trial 125 finished with value: 9.28444031232264e-06 and parameters: {'feature_weighting_width': 0.04056676166180355, 'period': 1.8802788277127025}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:51:59,978] Trial 126 finished with value: 7.093977486975935e-05 and parameters: {'feature_weighting_width': 0.01981371724571964, 'period': 1.8814079133487782}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,063] Trial 127 finished with value: 7.778948592416158e-05 and parameters: {'feature_weighting_width': 0.04391270774612462, 'period': 1.8794798834961999}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,131] Trial 128 finished with value: 2.746119244218937e-05 and parameters: {'feature_weighting_width': 0.017465587684002137, 'period': 1.8799326684166644}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,192] Trial 129 finished with value: 0.0003416942549413106 and parameters: {'feature_weighting_width': 0.06964861091541022, 'period': 1.8783118232567189}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,256] Trial 130 finished with value: 0.0010444202255232942 and parameters: {'feature_weighting_width': 0.38667637145760597, 'period': 1.881912217233372}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,322] Trial 131 finished with value: 8.81734322096895e-06 and parameters: {'feature_weighting_width': 0.10418671869906047, 'period': 1.8805243014726887}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,383] Trial 132 finished with value: 1.998253974127664e-05 and parameters: {'feature_weighting_width': 0.08644121279872516, 'period': 1.8808794999044365}. Best is trial 113 with value: 7.275040351762602e-06.
[I 2025-02-07 15:52:00,475] Trial 133 finished with value: 6.674034608141374e-06 and parameters: {'feature_weighting_width': 0.03989033027362253, 'period': 1.8804753725020271}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,546] Trial 134 finished with value: 4.157043349792861e-05 and parameters: {'feature_weighting_width': 0.09953397551339424, 'period': 1.8811455029017177}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,605] Trial 135 finished with value: 0.004538904344770736 and parameters: {'feature_weighting_width': 0.6354384196956347, 'period': 1.8804727990751835}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,669] Trial 136 finished with value: 4.6804098643932826e-05 and parameters: {'feature_weighting_width': 0.06456662574205706, 'period': 1.8797291712888164}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,724] Trial 137 finished with value: 0.00012154218447209077 and parameters: {'feature_weighting_width': 0.024626652429429567, 'period': 1.8817259753231452}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,788] Trial 138 finished with value: 1.872067672959691e-05 and parameters: {'feature_weighting_width': 0.1426027525708945, 'period': 1.88071298709614}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,848] Trial 139 finished with value: 0.0002531262413762275 and parameters: {'feature_weighting_width': 0.11044095464490934, 'period': 1.8823105048746445}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,912] Trial 140 finished with value: 0.0001146756703465301 and parameters: {'feature_weighting_width': 0.05138555972891829, 'period': 1.8792495762581123}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:00,970] Trial 141 finished with value: 9.879851963552178e-06 and parameters: {'feature_weighting_width': 0.035153161706530836, 'period': 1.8802562524933784}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,029] Trial 142 finished with value: 2.0072460793423035e-05 and parameters: {'feature_weighting_width': 0.001778298972802246, 'period': 1.880037419212915}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,118] Trial 143 finished with value: 3.976241773021079e-05 and parameters: {'feature_weighting_width': 0.0798845919011135, 'period': 1.881134779749404}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,195] Trial 144 finished with value: 7.606412186993635e-06 and parameters: {'feature_weighting_width': 0.047952029608264196, 'period': 1.8805731039440887}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,262] Trial 145 finished with value: 1.0655017785259667e-05 and parameters: {'feature_weighting_width': 0.07961900667534816, 'period': 1.8806774341467778}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,332] Trial 146 finished with value: 6.54532121439081e-05 and parameters: {'feature_weighting_width': 0.05604589082734568, 'period': 1.8813650715390593}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,402] Trial 147 finished with value: 7.617340419563688e-05 and parameters: {'feature_weighting_width': 0.11504283725366314, 'period': 1.8795089047771236}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,468] Trial 148 finished with value: 0.0001941186325440958 and parameters: {'feature_weighting_width': 0.1013530329656652, 'period': 1.8788642264398507}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,550] Trial 149 finished with value: 3.126960989512819e-05 and parameters: {'feature_weighting_width': 0.13833574308400226, 'period': 1.8799705898459127}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,627] Trial 150 finished with value: 7.671241566816736e-06 and parameters: {'feature_weighting_width': 0.031149455930896658, 'period': 1.8805899553364398}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,696] Trial 151 finished with value: 7.736479356037613e-06 and parameters: {'feature_weighting_width': 0.04325811865227881, 'period': 1.8805857276033775}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,763] Trial 152 finished with value: 1.8000811039063383e-05 and parameters: {'feature_weighting_width': 0.034799349205534305, 'period': 1.8808625580628764}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,812] Trial 153 finished with value: 9.185038304133034e-06 and parameters: {'feature_weighting_width': 0.022457497862144203, 'period': 1.8806577825065258}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,874] Trial 154 finished with value: 9.179567294533743e-05 and parameters: {'feature_weighting_width': 0.06948087730827297, 'period': 1.8815479694848114}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,935] Trial 155 finished with value: 3.303180863588584e-05 and parameters: {'feature_weighting_width': 0.04703672384288188, 'period': 1.8798677636417833}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:01,998] Trial 156 finished with value: 3.607519581593162e-05 and parameters: {'feature_weighting_width': 0.015246865236884524, 'period': 1.8811037190141602}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,062] Trial 157 finished with value: 8.026690386281908e-06 and parameters: {'feature_weighting_width': 0.05505643463691488, 'period': 1.8803447691569544}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,124] Trial 158 finished with value: 7.927163249771694e-05 and parameters: {'feature_weighting_width': 0.04872388367962511, 'period': 1.8794699251068687}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,195] Trial 159 finished with value: 8.314267197694706e-06 and parameters: {'feature_weighting_width': 0.0897047207858622, 'period': 1.8805520428215785}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,279] Trial 160 finished with value: 0.00010027730237115732 and parameters: {'feature_weighting_width': 0.0034503944045608387, 'period': 1.8816030342990937}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,346] Trial 161 finished with value: 8.042466667949059e-06 and parameters: {'feature_weighting_width': 0.08743150139720593, 'period': 1.8805352635026584}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,413] Trial 162 finished with value: 2.234513955190028e-05 and parameters: {'feature_weighting_width': 0.08110062679725157, 'period': 1.8809198252408885}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,474] Trial 163 finished with value: 1.7493616014404112e-05 and parameters: {'feature_weighting_width': 0.05201437444803749, 'period': 1.8800860241990498}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,533] Trial 164 finished with value: 7.503948567122963e-06 and parameters: {'feature_weighting_width': 0.08035207338439401, 'period': 1.8804510803954368}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,600] Trial 165 finished with value: 5.9483080477282064e-05 and parameters: {'feature_weighting_width': 0.03894324071110831, 'period': 1.8813193703941415}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,678] Trial 166 finished with value: 4.554748161854345e-05 and parameters: {'feature_weighting_width': 0.06756370228926804, 'period': 1.8797414806807986}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:02,741] Trial 167 finished with value: 7.105149688619408e-06 and parameters: {'feature_weighting_width': 0.02158097346332306, 'period': 1.880555631298144}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,058] Trial 168 finished with value: 0.00016987235012848168 and parameters: {'feature_weighting_width': 0.02256031779433454, 'period': 1.881969169715892}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,257] Trial 169 finished with value: 1.8363746569420024e-05 and parameters: {'feature_weighting_width': 0.0011366128892037058, 'period': 1.8808698989846748}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,355] Trial 170 finished with value: 0.009709507046039907 and parameters: {'feature_weighting_width': 0.9350630209287223, 'period': 1.8801204989872207}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,495] Trial 171 finished with value: 8.151171833951255e-06 and parameters: {'feature_weighting_width': 0.0891106368530812, 'period': 1.8805397208445118}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,584] Trial 172 finished with value: 7.515967127021868e-06 and parameters: {'feature_weighting_width': 0.0327853051344661, 'period': 1.880580167553924}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,684] Trial 173 finished with value: 3.211977355325056e-05 and parameters: {'feature_weighting_width': 0.03122345579055674, 'period': 1.881059167759034}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,808] Trial 174 finished with value: 1.1757646678382709e-05 and parameters: {'feature_weighting_width': 0.05159068612206965, 'period': 1.880726883607964}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:03,898] Trial 175 finished with value: 4.458221792522531e-05 and parameters: {'feature_weighting_width': 0.08206688032294689, 'period': 1.879753827450629}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:04,007] Trial 176 finished with value: 7.790872844769857e-06 and parameters: {'feature_weighting_width': 0.03402021097052517, 'period': 1.8805952018711836}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:04,070] Trial 177 finished with value: 8.799798889790778e-05 and parameters: {'feature_weighting_width': 0.021162827142182816, 'period': 1.8815256734872146}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:04,132] Trial 178 finished with value: 8.02140634237099e-06 and parameters: {'feature_weighting_width': 0.058030741754736284, 'period': 1.8805872758542626}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:04,197] Trial 179 finished with value: 2.1046862135697937e-05 and parameters: {'feature_weighting_width': 0.0401476670890612, 'period': 1.88002404768786}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:04,727] Trial 180 finished with value: 0.00012269743012590119 and parameters: {'feature_weighting_width': 0.020992160079224755, 'period': 1.8792041766312972}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:05,109] Trial 181 finished with value: 7.870168187016544e-06 and parameters: {'feature_weighting_width': 0.06234340314561043, 'period': 1.8805729352563416}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:05,485] Trial 182 finished with value: 3.597365183633661e-05 and parameters: {'feature_weighting_width': 0.05881401756832788, 'period': 1.8810988798115045}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:05,629] Trial 183 finished with value: 1.0492131346553677e-05 and parameters: {'feature_weighting_width': 0.06169461695704429, 'period': 1.880685118678601}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:05,713] Trial 184 finished with value: 1.204607615891037e-05 and parameters: {'feature_weighting_width': 0.03582706112146235, 'period': 1.8801951998376503}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:05,954] Trial 185 finished with value: 5.524298251893498e-05 and parameters: {'feature_weighting_width': 0.06848418364090358, 'period': 1.881281262587102}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,017] Trial 186 finished with value: 6.750805136988451e-06 and parameters: {'feature_weighting_width': 0.0015498023496325272, 'period': 1.8805220921347878}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,087] Trial 187 finished with value: 4.4152271871444154e-05 and parameters: {'feature_weighting_width': 0.025405859168333142, 'period': 1.8797501438199664}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,165] Trial 188 finished with value: 2.64243070469407e-05 and parameters: {'feature_weighting_width': 0.005594123098095143, 'period': 1.8809893996523845}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,234] Trial 189 finished with value: 1.6846303409151218e-05 and parameters: {'feature_weighting_width': 0.04305967972493898, 'period': 1.880095205711173}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,294] Trial 190 finished with value: 0.0001410028293843987 and parameters: {'feature_weighting_width': 0.016729240086484132, 'period': 1.8818290113765634}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,361] Trial 191 finished with value: 8.607874669847676e-06 and parameters: {'feature_weighting_width': 0.05368663367156619, 'period': 1.880620849601346}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,424] Trial 192 finished with value: 6.842098441502103e-06 and parameters: {'feature_weighting_width': 0.032767759614498404, 'period': 1.8804089203100232}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,483] Trial 193 finished with value: 8.924020474182813e-06 and parameters: {'feature_weighting_width': 0.03215143390322838, 'period': 1.880288802365785}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,542] Trial 194 finished with value: 1.3174146765606102e-05 and parameters: {'feature_weighting_width': 0.0020735033938333453, 'period': 1.8807689522876634}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,611] Trial 195 finished with value: 5.976221885556164e-05 and parameters: {'feature_weighting_width': 0.06668855909802168, 'period': 1.8796171979942433}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,684] Trial 196 finished with value: 5.475032626060776e-05 and parameters: {'feature_weighting_width': 0.04328845270585912, 'period': 1.8812798915591857}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,747] Trial 197 finished with value: 8.32307470358414e-06 and parameters: {'feature_weighting_width': 0.02610442915695301, 'period': 1.8803119548661666}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,815] Trial 198 finished with value: 2.665316338616854e-05 and parameters: {'feature_weighting_width': 0.055585202446630975, 'period': 1.880988137623733}. Best is trial 133 with value: 6.674034608141374e-06.
[I 2025-02-07 15:52:06,871] Trial 199 finished with value: 2.3983340546710644e-05 and parameters: {'feature_weighting_width': 0.07489475951296347, 'period': 1.8799895306521908}. Best is trial 133 with value: 6.674034608141374e-06.
In [5]:
print(joint_tuner)
Tuner: feature_weighting_width: 0.03989033027362253 period: 1.8804753725020271
In [6]:
fwls.set_X_kwargs(update=True, **joint_tuner.best_hyperpars)
fwls.set_X_train(data.predictors)
fwls.train(data.targets)
Optimal model¶
At this point, we have our optimal model. Now, we want to determine how robust this model is. Normally, if we are using an ordinary or weighted least-squares regressor, we can use the standard errors formula to get confidence intervals on our model coefficients. However, the feature weighting that we've introduced prevents us from doing so because it biases our model. Instead, we have to use sampling methods. To do this, puffins has three methods for estimating the uncertainties on regression coefficients.
These three methods are: i) jackknifing, ii) Monte Carlo simulations, and iii) Markov Chain Monte Carlo simulations.
Jack-knifing¶
Broadly speaking, jackknifing is a method of resampling our data to get an empirical estimate of the variance of model parameters. There are several flavours of jackknifing that we can use, with slight differences in their applications.
First, we have traditional jackknifing, or delete-1 jackknifing, also known as Leave-One-Out Cross-Validation (LOOCV). LOOCV entails systematically deleting one data-point at a time and re-running the model fit to the remaining data, then estimating the variance with the following formula:
$\widehat{Var}_{LOOCV} = \frac{n-1}{n} \sum_{i=1}^n \left( \hat{\theta}_{i} - \bar{\theta} \right)^2$,
where $n$ is the number of datapoints, $\theta$ is the j-dimensional estimator (model parameter vector or statistic), and $\bar{\theta}$ is the average value of $\theta$ across all $i$ jackknife iterations (Efron & Tibshirani, 1993).
For large $n$ datasets, doing a full LOOCV run might be computationally prohibitive. In light of that, clever statisticians have developed other versions of jackknifing that account for removing $d$ points at a time. This form of jackknifing is creatively referred to as delete-d jackknifing. The variance from delete-d jackknifing is given by:
$\widehat{Var}_{DdJK} = \frac{n-d}{d \dot m} \sum_{i=1}^m \left( \hat{\theta}_{i} - \bar{\theta} \right)^2$,
where $d$ is the number of data points that we delete per iteration, and $m$ is the total number of ways that we can delete $d$ out of $n$ datapoints, e.g. $\binom{n}{d}$ (Shao & Tu, 1995). Alternatively, if $m$ is prohibitively large, one can take a sufficiently large $\tilde{m}$ that enables the estimator to be a reasonable representation the resulting parameter space.
Again, $m=\binom{n}{d}$ might be a prohibitively large number of iterations to run depending on your dataset, so we now come to Grouped Jackknifing. In Grouped Jackknifing, we split the data evenly into G groups and refit our model to each Group. The variance for Grouped Jackknifing is given by:
$\widehat{Var}_{GJK} = \frac{G-1}{G} \sum_{i=1}^G \left( \hat{\theta}_{i} - \bar{\theta} \right)^2$,
Where G is the number of groups (Miller, 1964).
In all cases, we can construct confidence intervals (CIs) using the standard formulae drawing critical values from either a Normal or t-score distribution.
A comment on the construction of confidence intervals¶
While in the case of large n datasets the critical values for the t-disribution converges to the Normal distribution, in the case of smaller $n$ the t-distribution has much larger tails to reflect the uncertainty of using fewer datapoints. As such, the critical value for 95% CIs is given as $\approx2.26$ for a t-distribution and $\approx1.96$ for small n, resulting in larger CIs when using the t-distribution.
Monte Carlo simulations¶
Monte Carlo simulations offer a way of empirically estimating the variance of model parameters by repeatedly sampling your data under some assumptions. At each simulation or realisation of the data, you re-fit your model and then analyse the resulting distribution of your model parameters. This approach is particularly useful when analytical variance calculations are difficult or even inappropriate due to non-linearity in your model, or non-Normally distributed uncertainties on your data. In this use case, we follow the implementation from the Numerical Recipes in C textbook.
Markov Chain Monte Carlo¶
Taking this one step further, we can wrap our model into a Bayesian framework, whereby we assume that we have some prior knowledge of our model parameters that we want to incorporate into our analysis. Bayes' theorem is given by:
$P\left( \theta | d \right) = \frac{ P\left( d | \theta \right) P\left( \theta \right) }{ P\left( d \right)}$,
where $d$ is our data, $\theta$ are our model parameters that we want make inferences of, $P\left( \theta \right)$ is the prior information on our model parameters, $P\left( d | \theta \right)$ is the likelihood function (sometimes written as $\mathcal{L}\left( d | \theta \right)$), $P\left( \theta | d \right)$ is the posterior distribution of our model parameters, and $P\left( d \right)$ is effectively a normalisation factor. In plain words, the likelihood function is a way of estimating how likely the data are, given, or assuming a generative model with parameters $\mathcal{M}\left( \theta \right)$, while our posterior distribution is a represenation of how likely a given set of parameters $\theta$ is, given the data $d$. In our case, the likelihood function is the OLS, Ridge regression, or FWLS.
Technically, if both your data and model are well behaved, there is an analytical solution to this. However, in reality, this is seldom the case. Instead, there is a whole industry that develops numerical sampling methods to approximate the posterior distribution! In this code, we make use of the Affine Invariant Ensemble Sampler emcee (Foreman-Mackey et al., 2013).
We derive empirical confidence intervals from the data in the form of $\alpha$% Highest Posterior Density intervals.
Comparison¶
puffins allows for the user to instantiate an UncertaintyEstimator class and use any or all of the methods described above to estimate parameter variances and confidence intervals! Below, we'll run a Grouped Jackknife example, a Monte Carlo example, and a Markov Chain Monte Carlo (MCMC) example.
In [7]:
esimator = UncertaintyEstimator(data, fwls)
In [8]:
## Grouped Jackknife
n_groups = 50
n_delete = 5000# 15000/n_groups
gjk = esimator.run_grouped_jackknife_sampling(n_groups=n_groups, n_delete=n_delete)
Grouped Jackkinfe Sampling: 100%|██████████| 50/50 [00:04<00:00, 11.40it/s]
In [9]:
## Delete-d Jackknife
n_groups = 1000
n_delete = 8000
ddjk = esimator.run_delete_d_jackknife_sampling(n_groups=n_groups, n_delete=n_delete)
Delete-d Jackkinfe Sampling: 100%|██████████| 1000/1000 [00:30<00:00, 33.32it/s]
In [10]:
## Monte Carlo
mcs = esimator.run_monte_carlo_sampling(target_uncertainties=abs(flux_err),n_samples=1000)
Monte Carlo Sampling: 100%|██████████| 1000/1000 [00:34<00:00, 28.76it/s]
In [11]:
## Markov Chain Monte Carlo
## It's like 95% implemented, I just haven't pushed it yet.
In [12]:
def get_delete_d_jk_var(
n_data: int,
estimators: np.ndarray,
n_groups: int,
n_delete: int,
)-> np.ndarray:
mean_ = np.mean(estimators, axis=0)
lead = (n_data-n_delete)/(n_delete*n_groups)
var_ = lead * np.nansum((estimators - mean_)**2, axis=0)
return var_
In [ ]:
In [13]:
import arviz as az
fig, axes = plt.subplots(4, 1, figsize=(4, 12))
# Extract the relevant coefficients
gjk_coefs = gjk['sampled_coefs'][:, 1:5]
mcs_coefs = mcs['sampled_coefs'][:, 1:5]
ddjk_coefs = ddjk['sampled_coefs'][:, 1:5]
for i in range(4):
az.plot_density([gjk_coefs[:, i], mcs_coefs[:,i], ddjk_coefs[:,i] ], ax=axes[i], colors=['darkorange', 'dodgerblue','firebrick'], data_labels=['G-JK', 'MC','DD-JK'], hdi_prob=0.95)
axes[i].set_xticklabels([f'{x:.4g}' for x in axes[i].get_xticks()])
axes[i].tick_params(axis='x', rotation=45)
plt.show()
/var/folders/0t/dn2hb3px135gyc8xpdtmcx080000gp/T/ipykernel_93867/2840708348.py:12: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
axes[i].set_xticklabels([f'{x:.4g}' for x in axes[i].get_xticks()])
/var/folders/0t/dn2hb3px135gyc8xpdtmcx080000gp/T/ipykernel_93867/2840708348.py:12: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
axes[i].set_xticklabels([f'{x:.4g}' for x in axes[i].get_xticks()])
/var/folders/0t/dn2hb3px135gyc8xpdtmcx080000gp/T/ipykernel_93867/2840708348.py:12: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
axes[i].set_xticklabels([f'{x:.4g}' for x in axes[i].get_xticks()])
/var/folders/0t/dn2hb3px135gyc8xpdtmcx080000gp/T/ipykernel_93867/2840708348.py:12: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
axes[i].set_xticklabels([f'{x:.4g}' for x in axes[i].get_xticks()])
We can see from this that the Grouped Jackknife results in the smallest errors, the Delete-d has a larger distribution of parameter values, and the Monte Carlo simulations have larger errors. Of course in this case, the uncertainties on the data that we used are synthetic, but this pattern will generally hold as we'll see with another example.
In [14]:
import seaborn as sns
mcs_standard_betas = abs(mcs['coefs_mean'][1:]) / np.sqrt(mcs['coefs_var'][1:])
gjk_standard_betas = abs(gjk['coefs_mean'][1:]) / np.sqrt(gjk['coefs_var'][1:])
ddjk_standard_betas = abs(ddjk['coefs_mean'][1:]) / np.sqrt(ddjk['coefs_var'][1:])
plt.figure(figsize=(10, 6))
sns.histplot(mcs_standard_betas, bins=range(0,50), fill=False, hatch='//', kde=False)
sns.histplot(gjk_standard_betas, bins=range(0,50), fill=False, hatch='\\', kde=False)
sns.histplot(ddjk_standard_betas, bins=range(0,50), fill=False, hatch='||', kde=False)
plt.axvline(3, color='red', linestyle='--')
plt.title('KDE of Standard Betas')
plt.xlabel('Standard Betas')
plt.ylabel('Density')
plt.show()
Finally, here we see that most of the regression coefficients are insignificant at the $3-\sigma$ level no matter how you calculate your $\mathrm{Var}\left(\beta\right)$'s. We could use this information to remove the "insignificant" coefficients, or we can leave them in knowing that the feature weighting is going to make these coefficients not have a significant contribution to the model anyways.
One of the major benefits of doing an analysis of this is that we can propagate the uncertainties to our predictions, which we'll do below.
In [15]:
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
t_pred = np.linspace(0, 2*period, 1000)
gjk_p1 = fwls.predict(t_pred, coefficients=gjk['coefs_mean'] + 3 * np.sqrt(gjk['coefs_var']))
gjk_m1 = fwls.predict(t_pred, coefficients=gjk['coefs_mean'] - 3 * np.sqrt(gjk['coefs_var']))
mcs_p1 = fwls.predict(t_pred, coefficients=mcs['coefs_mean'] + 3 * np.sqrt(mcs['coefs_var']))
mcs_m1 = fwls.predict(t_pred, coefficients=mcs['coefs_mean'] - 3 * np.sqrt(mcs['coefs_var']))
ddjk_p1 = fwls.predict(t_pred, coefficients=ddjk['coefs_mean'] + 3 * np.sqrt(ddjk['coefs_var']))
ddjk_m1 = fwls.predict(t_pred, coefficients=ddjk['coefs_mean'] - 3 * np.sqrt(ddjk['coefs_var']))
ax.fill_between(t_pred, y1=gjk_m1[1], y2=gjk_p1[1], color='darkorange', label='G-JK')
ax.fill_between(t_pred, y1=mcs_m1[1], y2=mcs_p1[1], color='dodgerblue', label='MC')
ax.fill_between(t_pred, y1=ddjk_m1[1], y2=ddjk_p1[1], color='firebrick', label='Dd-JK')
ax.set_xlabel(r'${\rm Forecasted~Time}$')
ax.set_ylabel(r'${\rm Predicted~Flux}$')
ax.legend()
fig.tight_layout()
In [ ]: