admin/lib/tariff behavior changes with 8.2.42xx

[catsmanac]

Settings for battery modes and their behavior for charging and discharging are shown in the admin/lib/tariff endpoint. Some settings in the Envoy can be changed using the Admin/lib/tariff endpoint. Updating fields in Tariff.StorageSettings and using a PUT with only the Tariff object allows control of Envoy settings.

This method can be used to change mode to either self-consumption, full backup or savings, change from grid or set reserve soc and so on.

The better word seems to be could. With the roll-out of firmware version 8.2.42xx or newer, behavior has changed. Changing these settings from the Enlighten App or web-site works, but changed settings are no longer reflected in the admin/lib/tariff endpoint. Changing the controls through the API still updates endpoint settings, but the Envoy ignores these changes.

Not clear if there's alternative endpoints for thus in the newer firmware.

[erikeidt]

@catsmanac

I'm using HA to PUT tariff.json, mornings to turn on ZN, to change peak times (due to peak bug), to activate DG dynamically toward the end of peak.

I got updated to D8.2.4378 a few days ago. (I'm on NEM2PS; I don't see the dreaded AI Optimization, just the usual Self-Consumption, Savings, and Full Backup modes in System Profile.)

peak_rule=DG (with "enable_discharge_to_grid":true) no longer works. Watching it, it appears to try for a split second, but then reverts to DL behavior. Almost like it is trying but immediately hits some limit. I'm searching to find another setting that needs to be enabled to allow DG but haven't as yet.

When I apply a tariff using DG the envoy accepts it, and also re-updates the schedule object but seems to make an off by 1 mistake in counting the "duration" (reminiscent of tariff corruption) — this off by 1 doesn't happen when PUT of tariff.json using peak_rule DL or ZN.

peak_rule=ZN still working, as well as changing schedules via PUT to tariff.json.

I use ZN in the mornings because I don't want to pull power from the grid when solar is available, so I exit CP (off-peak) mode and use peak_rule=ZN when there's enough solar to power the home load and then charge batteries from excess solar (i.e. after the home load is powered). (I don't use mode=self-consumption because I've had too many tariff corruptions when using that). If the solar output drops below a set level, I revert back to off-peak CP with HA.

I change schedules because the bugs in the App/Web UI require specifying a peak even when you don't have one, as in my case, there is no peak on weekends, but we cannot say that (or run afoul of more tariff corruption problems). So, I have to put a fake peak some time of the day on weekends, and then when that time comes, I change to tariff to having a different (earlier) peak time — that way the system seems happy with the tariff, but we dodge the fake peak.

There's also a new "opt_schedules":false in the storage settings object that I don't know what it's for.

@Matthew1471

"very_low_soc" is the off-grid battery shutdown setting; it is the one found via Settings->Battery "Battery shutdown level", whereas "reserved_soc" is the on-grid battery stop limit for DL/DG discharge found in Profile selection. When off grid, powering the home load via discharge is limited by the former, and so the system will shutdown until solar comes on in the morning. The purpose seems to be to hold enough battery in reserve to restart the system the next day.

[catsmanac]

@erikeidt, thanks for the additional details.

There's also a new "opt_schedules":false in the storage settings object that I don't know what it's for.

Yes, we found that too and there's a patch pending to include it in the PUT when it exists in the tariff.json.

So with the firmware D8.2.4378 there's some change to what you described before, but not that much it doesn't work at all anymore.

peak_rule=DG (with "enable_discharge_to_grid":true) no longer works.
When I apply a tariff using DG the envoy accepts it,
peak_rule=ZN still working, as well as changing schedules via PUT to tariff.json.
If the solar output drops below a set level, I revert back to off-peak CP with HA.

For the record, you are sending json files with only the (whole) tariff section changes from nan existing one, using an HA automation calling Curl system command. You are not using any of the HA Enphase integration actions for switches, selectors or numbers?

What is the difference between the first 2 items: peak_rule=DG (with "enable_discharge_to_grid":true) <-> When I apply a tariff using DG the envoy accepts it,

[erikeidt]

@catsmanac

For the record, you are sending json files with only the (whole) tariff section with changes from an existing one, using an HA automation calling Curl system command.

Yes, for tariff changes. I statically pre-construct 4 different tariff.json's and choose between them dynamically using HA automations using the following from HA Enphase integration:

You are not using any of the HA Enphase integration actions for switches, selectors or numbers?

For Automations & dashboards:
sensor.envoy_xxxx_current_power_production (Solar Production)
sensor.envoy_xxxx_battery (Battery State of Charge %)
select.enpower_xxxx_storage_mode (Savings/Self-Consumption/Full Backup)
number.enpower_xxxx_reserve_battery_level (Reserve Battery Level***)

For dashboards only (at present)
sensor.envoy_xxxx_current_power_consumption (Home Load)
sensor.envoy_xxxx_current_net_power_consumption (Grid Usage)
sensor.envoy_xxxx_current_power_consumption_l1/l2 (CT1 & CT2)

*** I set a different reserve battery level for each of my 4 tariffs. I use this to quickly determine which tariff has been taken by my Enphase system. I would also like to see the tariff.schedule.date object exposed (as a sensor?) since I believe that captures the time of the last PUT to tariff.json (as it seems to automatically rebuild that tariff.schedule object upon PUT).

I'm currently using curl to PUT to tariff.json, and this has several drawbacks, so I'd prefer to write to tariff.json (from a string or file) using the Enphase integration, but I don't know how to do that as yet.

As far as drawbacks to using curl directly:

The PUT to tariff.json returns an up-to-date tariff.json, which I discard (as I don't want to parse that), and I assume if the Enphase integration was doing the PUT it would parse the returned tariff to immediately update the sensors, numbers, switches as appropriate.

That curl PUT I'm doing can fail, and when it does I don't do error handling in automation. I am therefore using curl command line options that retry automatically (e.g. every 2 seconds, up to 9 times). And still no error handling or detection if all 9 reties don't work.

---

What is the difference between the first 2 items: peak_rule=DG (with "enable_discharge_to_grid":true) <-> When I apply a tariff using DG the envoy accepts it,

I have been using peak_rule:DG, with enable_discharge_to_grid:true, and it accepts this PUT and returns a larger tariff object with schedule sub-object rebuilt as I would expect it to do. The problem is that it doesn't actually do any discharge to grid and actually acts like a weak DL (discharge to home load, but occasionally import 100-200w from the grid during peak, i.e. this is worse than plain DL). This is new behavior with D8.2.4378. I am wondering if they have changed some default discharge export limit, but I am not an installer so cannot check certain ivp settings.

The first few times I tried a DG tariff with the new D8 software, the web monitoring (https://enlighten.enphaseenergy.com/web/xxxx/history/graph/hours, Live Status) flickered as if it was trying to enter DG but was throttled immediately. Since then it doesn't even do that, just continues as if with (weak) DL after tariff switch.

With the old D8.2.127 software (which I had as late as last month) switching to DG tariff, I would quickly get to a steady discharge to the grid while also powering the home load using those same settings (peak_rule:DG and enable_discharge_to_grid).

I have also found that we can use peak_rule:DL with enable_discharge_to_grid:true and it will rebuild the schedule object having DG as the "setting:" in tariff.schedule.tariff..[].setting: for peak times (instead of DL).

I am going to experiment with some other combinations within tariff.json, but fundamentally suspect that there is something new at work beyond the tariff to prevent entirely or limit in some way discharge to the grid. (Given the flicker I had at first with the new software and that that's gone now, it could be a cumulative limit, say monthly or yearly? Rather than a instantaneous limit?)

When my system was first setup in late 2019, and using it in early 2020, it would discharge to the grid (the entire peak period, which was not very intelligent) so I know my system is physically capable of it and that it is/was legal given my utility agreement. With HA automation, what I'm trying to do is to use DL to power home load during (first part of) peak, and dynamically, if there's enough battery power left (based on how we have or haven't splurged consumption on battery during the first part of peak) to discharge some to grid toward the end of peak. Since the discharge rate is constant (is there an installer setting to control this?), we have to back up in time from end of peak, and that's where I'm applying HA automation.

Eventually, I'd like to get HA automation to also control our HVAC/AC in coordination with peak times & battery usage. I have indoor temp sensors visible in HA, but not a controller/thermostat as yet.

[catsmanac]

so I'd prefer to write to tariff.json (from a string or file) using the Enphase integration, but I don't know how to do that as yet.

That's currently not available, but surfaced as useful during this whole firmware shake-up and might get added. Although it would be an 'expert tool' to use and there may be risks with malformed ones. It would require thorough understanding of what to change.

In HA there's options to run python scripts as actions that might help here to pre-cook the set of data.

I would also like to see the tariff.schedule.date object exposed (as a sensor?)

We're currently getting date from Tariff and Tariff.storage_settings, but not from storage.

The question is also if all this will be a moot point when the next firmware hits the Envoy!?

[erikeidt]

@catsmanac

A bit more experimenting today:

• operation_mode_sub_type: "prioritize-energy" appears to be the same as System Profile -> Savings Mode -> "Use battery after peak hours" — this builds schedule: objects having initial setting:CP for before peak, settings:DL for during peak, and settings:ZN for after peak.

• opt_schedules seems to override any tariff provided schedule and substitute schedule.batt_mode & battery_mode=self-consumption with no real TOU schedule, even when mode=economy in the main tariff object. Not sure how that's useful, since its just self-consumption with no schedule, but I guess maybe it keeps your main tariff TOU information so you can revert to economy/savings, unlike mode=self-consumption where your selected tariff may be wiped out.

---

enable_discharge_to_grid:true makes peak_rule:DL turn into settings:DG, but have yet to see actual discharge to grid on my system. Still looking elsewhere.

As I mentioned earlier, have not found any approach that will allow once again discharge to grid, so still thinking some other setting is standing in the way.

---

The question is also if all this will be a moot point when the next firmware hits the Envoy!?

I know, super frustrating to have something working nicely and broken with no notice or known workaround by a firmware/software update.

I'm also looking into taking the coursework to gain installer permission over my system; don't know if it will help, though even if it bears fruit, surely this would be more than the average HA user could be expected to do.

[erikeidt]

@catsmanac

With the roll-out of firmware version 8.2.42xx or newer, behavior has changed. Changing these settings from the Enlighten App or web-site works, but changed settings are no longer reflected in the admin/lib/tariff endpoint. Changing the controls through the API still updates endpoint settings, but the Envoy ignores these changes.

As of Envoy firmware 8.2.4225, the Envoy no longer supports setting battery modes, enabling/disabling charging from the grid, or changing reserve battery level through the local REST API used by HA. Until a resolution is found, you will need to use the Enphase APP to control these.

I'm on 4.2.4378 since Dec 5th. The following functions are still working for me via HA/curl: set tariff including battery mode, though I only use savings mode; if I want self-consumption behavior, I use savings mode with peak_rule=ZN, if I want full backup behavior I set the reserve battery level to 100%.

As I mentioned what's not working anymore for me is "enable_discharge_to_grid: true". Though continues to build a proper "schedule:" object in the tariff with "schedule.tariff[].setting:DG" (as before with D8.2.127), but the system does not honor the discharge to grid, in fact pulls some power (between 0 and 200w) from the grid instead. I will try working with my installer to see if something else has been changed. Support at Enphase is baffled and has no answer.

Best,
Erik

P.S. looking forward to enphase_envoy.post_data

[catsmanac]

Thanks for all that testing @erikeidt.

F.w.i.w., I found this:

Adds a discharge battery to the grid for an export-only system running on the Self Consumption profile to increase savings under NEM 3.0.

in the release notes for 8.2.4287. Maybe not related as it's for NEM 3.0

[joergbattermann]

Thanks for all that testing @erikeidt.

F.w.i.w., I found this:

Adds a discharge battery to the grid for an export-only system running on the Self Consumption profile to increase savings under NEM 3.0.

in the release notes for 8.2.4287. Maybe not related as it's for NEM 3.0

FYI @erikeidt : I've seen & tried and then asked Enphase about that functionality a few months back when they rolled the FW out and that functionality is.. quite something (logically speaking).

It is available for (California) NEM 3 sites only afaik and you can't control when it's exporting but it automatically starts dumping all your stored energy (down to your configured reserve level) every time the $/kWh rate of exported kWhs exceeds the cost of $ per kWh imported. It doesn't do any forecasting of your household's load and/or if there's a 'more lucrative' time later in the day, its only trigger is a simple comparison of the hourly import vs export rate / kWh and it checks that at the beginning of every hour of each day. See how it looks like in Enlighten.. just a on/off switch:

So it's a rather blunt, rudimentary functionality that's not really doing a hell lot other than blindly exporting all you got stored the moment export rate is a hair more than import rates.

[catsmanac]

That's indeed pretty rudimentary @joergbattermann. And driven from their cloud and prices it's getting from somewhere else I assume?

As for the enphase_envoy.post_data, @erikeidt, first step is still waiting for approval. Had to change it as it was to large to implement in one go. Then a PR for the actual post needs to follow.

[joergbattermann]

That's indeed pretty rudimentary @joergbattermann. And driven from their cloud and prices it's getting from somewhere else I assume?

Yeah in Enlighten / the cloud you select (or configure manually) your utility import and the hourly changing export rates (every hour 'window' of every day has potentially different export rates under NEM3), ToU windows and that is used to control if/when it exports. These same values are used for AI Optimization as well.

My utility (Pacific Gas & Electric / PG&E + a local CCA (MCE)) and all other California utilities had and will have to have their NEM3 export rates approved and they need to make these available (albeit make it hard for end customers to find the raw data), but you can have a look at i.e. pge.com/eecvalues.

Given the export rates' dataset is so enormous (relatively speaking to normal ToU tariffs), I'd assume it's one reason why Enphase didn't want these to exist on your local Envoy and its tariff.json anymore but make the FW change to have the Envoy check with their cloud every hour for the current value(s).

See i.e. the Export rates today:

vs next week:

vs next month (and so on):

.. vs the more static Import ones:

and Winter:

Anyway, mind you the stark difference between import vs export rates that we get - this time of the year we pay 6-10x more per kWh imported vs what we get exported, and the gap widens even more in i.e. April when we get a lovely $0.00 / kWh exported. Zero.

The only two months when exporting actually does make sense and is somewhat 'lucrative' is August and September.. and that's probably why Enphase introduced AI Optimization just a day or two into August last year to have some good sales argument for potential future NEM3 customers. However, their cloud seemingly could not handle the demand and Enlighten broke down entirely during the 1-2h windows each day because (I assume) a) they set all NEM3 sites to AI Optimization automatically without asking home owners and therefore b) having a ton of Envoys making additional requests to their backend and c) the customers that were aware about that switch probably opened Enlighten more often than before to check IF it's actually doing something / exporting at the right time.

Anyway, Enlighten/their cloud ended up timing out quite often, Envoys didn't get a response and therefore didn't start exporting in the short 1-2h time window during which the export rates were high.

So not only was it a rushed, unfinished functionality that broke things (see our issues) but it was also forced onto people (automatically switching all sites into AI Optimization profile here) and then (subjectively) face planted when it didn't work because of the backend being undersized/poorly implemented (or whatever else contributed to the issues).

Anyway & TL;DR: yep.. import rates AND export rates are configured in the cloud / Enlighten ;)

[catsmanac]

With that explanation I now understand some of the comments I found at the time we were looking what was going on with this new mode. They were along what you describe, as well as many complaints on failures getting tokens, which would also be explained by that overload you describe.

[catsmanac]

@erikeidt, finally finished the post mechanism. In the end made it a custom integration, it requires definitely some expertise like yours to build to correct data to send. It can run side-by-side to the core or other custom integrations. See https://github.com/catsmanac/ha_enphase_envoy_raw_data#enphase-envoy-raw-data.