NightMare Networkv1.0.0-alpha.1

Search every page of the documentation. Or start with:

Docs / Protocols / Resource protocol

Resource protocol

Resources are the application-level contract between NightMare devices.

The current Resource protocol has these topic shapes:

<device>/manifest/msgpack
<device>/manifest/json
<device>/manifest/consume/msgpack
<device>/manifest/consume/json
<device>/resource/<name>/state
<device>/resource/<name>/set
<device>/resource/<name>/invoke
<device>/resource/<name>/event

The central rule is:

A manifest describes. /state tells the truth.

There are three kinds of Resource, and each one says something different:

Value  = something is           retained /state
Action = please do something    transient /invoke
Event  = something happened     transient /event

Protocol version#

The current Resource manifest version is:

6

The version appears in the retained manifest document. Version 6 adds the event kind; the value and action shapes are unchanged.

Resource names#

A Resource name is one MQTT topic segment.

It must:

  • contain 1 to 64 characters,
  • not contain /, +, or #,
  • not contain control characters below 0x20.

A device can bind at most:

100 Resources

A resolved (device, resource-name) address may only be represented once in the local Resource registry.

Manifest#

A device publishes its retained manifest at:

<device>/manifest/msgpack
<device>/manifest/json    optional when NM_ENABLE_JSON_WIRE=1

Only Resources Managed by that device appear in its manifest. Remote Resources are local references and are not announced as capabilities of the current device.

A representative optional JSON rendering is:

{
  "version": 6,
  "resources": [
    {
      "name": "temperature",
      "kind": "value",
      "access": "read",
      "type": "float",
      "advertise_ms": 300000,
      "hardware": {
        "poll_ms": 1000,
        "flags": 7,
        "connected": true,
        "note": "Address: 0x48"
      }
    },
    {
      "name": "power",
      "kind": "value",
      "access": "read_write",
      "type": "boolean",
      "advertise_ms": 300000
    },
    {
      "name": "ac_door",
      "kind": "value",
      "access": "read",
      "type": "boolean",
      "depends_on": "door",
      "advertise_ms": 300000
    },
    {
      "name": "set_timer",
      "kind": "action",
      "arguments": [
        {
          "name": "seconds",
          "type": "integer",
          "required": true
        }
      ]
    },
    {
      "name": "acoustic:beep",
      "kind": "event",
      "type": "integer"
    }
  ]
}

The manifest is retained.

The current manifest payload limit is:

16384 bytes

Canonical MessagePack manifest#

The canonical manifest is retained as MessagePack at:

<device>/manifest/msgpack

Its positional schema is:

[encodingVersion, manifestVersion, resources[]]

value  = [0, name, accessEnum, typeEnum, dependsOn|null,
          advertiseMs, hardware|null]
hardware = [pollMs, flags, connected?, note?]
action = [1, name, arguments[]]
arg    = [name, typeEnum, required]
event  = [2, name, typeEnum]

The leading number is NetResourceType: 0 value, 1 action, 2 event.

dependsOn is a single local Resource name. Hardware trailing elements are omitted; when a note exists without connection reporting, the connection slot is null. Actions have no advertisement or hardware policy. An Event has neither, and no access, dependency or arguments: it declares only its name and payload type.

Encoding version 3 is current. Version 2 replaced the former enabled/seconds positions; version 3 introduces the event shape. Array positions and numeric enums are append-only. Readers that do not recognize the encoding version reject the document. There is no compatibility fallback or old-format reader. The bare <device>/manifest root is a namespace and never carries a payload.

Consume manifest#

A separate retained document describes Remote Resources this device currently depends on:

<device>/manifest/consume/msgpack
<device>/manifest/consume/json    optional when NM_ENABLE_JSON_WIRE=1

This is not an expansion of the provider manifest. The consume manifest version is 4. remotes is the only representation: every declared Remote Resource is present once. bound: true derives an active consume edge; bound: false describes a configurable input with no edge. device and resource are empty while unbound. There is no serialized consumes array.

{
  "version": 4,
  "remotes": [
    {"name": "temperature", "bound": true,
     "device": "weather-node", "resource": "temperature",
     "kind": "value", "access": "read", "type": "float"},
    {"name": "door", "bound": false, "device": "", "resource": "",
     "kind": "value", "access": "read", "type": "boolean"},
    {"name": "beep", "bound": true,
     "device": "Watson", "resource": "acoustic:beep",
     "kind": "event", "type": "integer"},
    {"name": "chime", "bound": false, "device": "", "resource": "",
     "kind": "event", "type": "string"}
  ]
}

A Remote Event entry has no access: there is nothing to read or write.

The compact positional schema is:

[encodingVersion, consumeVersion, remotes[]]

arg    = [name, typeEnum, required]

remote value  = [0, localName, bound, device, resource, accessEnum, typeEnum]
remote action = [1, localName, bound, device, resource, arguments[]]
remote event  = [2, localName, bound, device, resource, typeEnum]

Encoding version 3 is current. Removing the old duplicated positional array is incompatible, so readers accept only version 3. The document is republished when a Remote Resource is bound, retargeted, detached, or unbound, and on reconnect. Its bare root is only a namespace.

Resource dependencies#

A Managed Value may declare that it is another Value:

acDoor.dependsOn(doorSensor);

This asserts that ac_door and door are the same logical value and that door is authoritative. Every time door receives an authoritative update, the Manager copies it into ac_door and publishes ac_door's state. No application code runs in between.

The declaration appears in the provider manifest as a singular depends_on:

{
  "name": "ac_door",
  "kind": "value",
  "access": "read",
  "type": "boolean",
  "depends_on": "door"
}

Local-name addressing#

depends_on names a local Resource, never <device>/<resource>. This is what keeps the two documents independent:

Mycroft/door            ManagedSensor<bool>, the authoritative observation
  |  source
Adler/door              RemoteSensor<bool>, a local handle for it
  |  dependsOn
Adler/ac_door           ManagedSensor<bool>, Adler's own representation

The manifest says ac_door depends on door. The consume manifest says door currently reads Mycroft/door. Retargeting door to Moriarty/door changes only the second document; nothing that mirrors door is touched.

A reader resolves the depends_on name against the same device: a Managed source is in that device's manifest, a Remote one in its consume manifest under remotes. A name that appears in neither was declared against a Resource that was never bound.

Withdrawal propagates#

A mirror cannot keep asserting a value once the source stops having one. When the source loses its value, each dependent becomes unavailable and its retained /state is tombstoned:

Mycroft/door/state tombstoned
    -> Adler/door      UNAVAILABLE
    -> Adler/ac_door   UNAVAILABLE, retained /state tombstoned

The decoded value stays readable locally. What is withdrawn is the claim that it is current, so a reconnect does not re-announce it either.

Without this, a consumer that does not resolve depends_on for itself would read Adler/ac_door/state as a perfectly valid retained value that nothing stands behind any more.

Three events withdraw:

the source's retained state is tombstoned
the source is retargeted or cleared with SOURCE
the source is unbound

The last two also discard what the source learned, so the dependent has nothing left to mirror.

Semantic identity is the application's claim#

dependsOn() asserts semantic identity, not merely causal dependency. The application author is responsible for declaring only Resources that represent the same logical value. NightMare and backend tooling may detect structural and type inconsistencies but cannot guarantee semantic equivalence.

The firmware checks only what it can see: a Value cannot depend on itself, and the two wire types must match.

bool    -> bool      OK
float   -> float     OK
integer -> integer   OK

bool    -> integer   rejected
float   -> string    rejected

Wire-type equality is not exact type equality: int8_t and uint32_t are both integer on the wire, so a mismatch between them is not caught.

occupied.dependsOn(doorOpen) passes every check the firmware can make and is still nonsense. Backend tooling is expected to report unresolved names, type mismatches, self-dependencies, and unsupported chains; semantic correctness remains the application's.

Limits of version 1#

One dependency per Value, and the last dependsOn() call wins:

a.dependsOn(b);
a.dependsOn(c);

// final relationship:
a -> c

ManagedSensor<T> only. Actions have no dependencies, and a Remote Value cannot declare one: it already mirrors its own source.

ManagedState<T> is deliberately excluded until write-through is designed. A mirror is read-only by construction, while a ManagedState accepts /set, and the consistent answer is for the write to travel to whoever is authoritative and return as an ordinary mirrored update:

SET dependent
    -> forward the request to the authoritative source
    -> source changes
    -> the normal mirror comes back

That does not exist yet, so nothing writable can declare a dependency and advertise a value it cannot change.

Propagation is one level. A dependent is not treated as a source in turn, so a chain a -> b -> c stops at b. The manifest publishes only the direct local edge and does not resolve the full path.

Chain resolution, source-of-truth discovery across devices, and automatic subscription redirection are future protocol features:

B -> A -> C

getChain(B)       = [B, A, C]
sourceOfTruth(B)  = C

A future implementation may redirect B from C to D when A is retargeted, while preserving B's configured source as A. None of that exists today.

Dependencies are firmware declarations. They are not persisted and cannot be configured at runtime.

Resource kinds#

kind is one of:

value
action
event

Values include access and type.

Actions include an arguments array.

Events include type, the type of the payload each occurrence carries, and nothing else. An Event is a distinct kind, not a Value with a flag: it has no access, depends_on, advertise_ms, hardware or arguments.

Value access#

The manifest encodes Value access as:

read
read_write

The standard C++ wrappers map to these policies:

ManagedSensor<T>   -> read
RemoteSensor<T>    -> expects a readable remote Value

ManagedState<T>    -> read_write
RemoteState<T>     -> expects a writable remote Value

A locally declared read-only Remote Value may observe a remote read_write Value. A local read_write Remote declaration requires the remote manifest to declare read_write for compatibility diagnostics.

Manifest compatibility is diagnostic only and does not gate state or requests.

Value types#

The wire taxonomy is:

string
boolean
integer
float
struct
time
colour

The built-in NetCodec<T> implementations currently cover:

std::string
bool
integral C++ types
floating-point C++ types
TimeType
ColourType

struct exists in the wire taxonomy and Action argument metadata, but there is no generic built-in NetCodec<T> that silently serializes arbitrary C++ structs.

Unsupported Value C++ types fail at the codec boundary rather than degrading to std::string.

Value state#

A Managed Value publishes its state at:

<device>/resource/<name>/state

The state is retained.

The payload is the encoded Value itself; it is not wrapped in a generic JSON object.

Boolean#

Encoding:

true
false

Decoding also accepts:

1
0

and case-insensitive true / false.

Integer#

Encoding is decimal text.

Examples:

0
42
-12

Decoding rejects trailing text and rejects values outside the C++ target type's range.

Unsigned types reject negative input.

Floating point#

Encoding uses a compact decimal representation that round-trips to the same float or double.

Examples may look like:

23.5
0.125
1.25e+06

Decoding uses the whole payload as one floating-point number; trailing text is rejected.

std::string#

The payload is the raw std::string.

Example:

cool

There is no JSON quoting added by the Resource protocol.

An empty std::string is not a valid Resource std::string value because the empty MQTT payload is reserved as the retained deletion marker.

Time#

TimeType represents local time of day only. Canonical encoding is:

HH:MM:SS

Decoding additionally accepts HH:MM, HHMMSS, HHMM, H:M:S, and H:M. Hours must be 0–23 and minutes/seconds 0–59. It carries no date, epoch, duration, timezone, or UTC meaning.

Colour#

ColourType packs RGBA as 0xRRGGBBAA; an RGB form implies alpha 255. The canonical wire encoding is the raw unsigned 32-bit value in decimal.

Decoding also accepts 0xRRGGBB, #RRGGBB, 0xRRGGBBAA, #RRGGBBAA, rgb(r,g,b), rgba(r,g,b,a), and hsv(h,s,v). Function components use the byte range 0–255; HSV hue wraps across the same byte range.

Appending time and colour to NetValueType preserves all earlier numeric enum values, so the compact manifest encoding version remains unchanged.

Every Managed Value declares one signed advertise_ms interval. Available Values with a non-negative interval publish changes immediately. 0 is event-driven only. A positive interval periodically reaffirms unchanged retained state under normal framework servicing; it is not a hard real-time guarantee. A negative interval suppresses publication; -1 is the canonical disabled value. Each successful publication resets the refresh timer.

Changing to a negative interval withdraws retained /state with an empty retained payload and suppresses later advertisements. Returning to a non-negative value publishes the authoritative value when one exists and the Resource is available. Policy changes are persisted before their runtime commit. A failed provider manifest publication remains dirty and is retried with rate limiting. Both the JSON and compact manifests must publish successfully before that dirty state is cleared, because Remote freshness consumes the compact metadata.

Availability is not added to ordinary payloads. A Managed owner represents temporary unavailability by withdrawing retained state while leaving the Resource bound and declared. Advertisement policy does not define polling, power, acquisition, or control-loop behavior.

Hardware policy#

Managed Values may opt into a hardware manifest object. poll_ms > 0 is the desired interval, 0 means firmware/system default behavior, and < 0 disables acquisition. Flags are capabilities: 1 reports connection state, 2 permits remote poll changes, and 4 permits remote disabling. connected is emitted only with flag 1 and defaults to true until firmware reports otherwise; note is optional informational UTF-8 limited to 64 bytes. NightMare does not perform polling. Only a remote poll_ms override is persisted.

State freshness#

Availability and freshness are separate runtime state.

A valid non-empty owner state makes the Remote Value available and:

FRESH

An empty /state payload means the retained state was deleted and makes the Remote Value unavailable. It does not itself set freshness to STALE.

While a Remote Value is available and its owner's advertisement policy is known and positive, housekeeping compares lastUpdateMs with the advertised period. At approximately twice that period without a valid reaffirmation it becomes:

STALE

Interval 0 disables time-based aging. A negative interval pauses policy-based aging and declares transport withdrawal only; it does not change Resource or hardware availability. Only /state traffic changes Value availability.

The last decoded value remains stored and readable through getValue(), but available() and hasValue() report that it is not current while unavailable. A later valid state immediately restores availability and FRESH.

Changing a Remote Resource's source is different: it resets the old source state entirely, including authoritative value presence, freshness, optimistic state, and update timestamps.

Before the first owner state is learned, freshness is:

UNKNOWN

Manifest metadata and freshness#

The manifest has a different lifecycle from Value state.

The manifest supplies the period used to age valid values, but receiving a manifest does not itself make a Value fresh. The following do not directly change Value freshness:

  • a missing manifest,
  • a withdrawn manifest,
  • a malformed manifest,
  • a manifest that does not contain the expected Resource,
  • a manifest type/access disagreement.

Likewise, manifest compatibility does not gate:

/state
/set
/invoke
/event

This separation lets state remain useful even when descriptive metadata is absent or temporarily inconsistent.

Remote manifest diagnostics#

When a manifest from a device used by local Remote Resources arrives, NightMare compares it with the local declarations.

For Values it checks:

  • Resource exists,
  • kind matches,
  • type matches,
  • if the local Remote Value is read_write, remote access is also read_write.

For Actions it checks that the Action exists and, when the local RemoteAction declares expected arguments, compares those expectations with the remote schema.

For Events it checks that the Event exists, that kind is event, and that the payload type matches. An Event has no access, so there is nothing further to compare.

Disagreements are logged. They do not rewrite the local declaration and do not disable traffic.

Value write request#

A writable Value receives requests at:

<device>/resource/<name>/set

The message is transient.

The payload uses the same codec as /state.

Example:

topic:
bedroom-ac/resource/target_temperature/set

payload:
23.5

The implementing device only subscribes to /set for Managed Values declared read_write.

A ManagedState<T> decodes the payload and calls its onWrite handler.

The handler returns:

true
    request accepted

false
    request rejected

If accepted, the requested Value becomes authoritative owner state and is published retained on /state.

If rejected or undecodable, state remains unchanged.

Local Managed Value writes#

Calling setValue() on a Managed Value changes local truth even when transport is unavailable.

NightMare attempts to publish the new retained /state best-effort. A later reconnect re-announces Managed Values that have authoritative state.

This reflects the ownership rule:

Network availability does not decide whether the owner itself knows its own state.

RemoteState writes#

Calling setValue() on a RemoteState publishes:

<owner>/resource/<name>/set

The call only reports success if the request was accepted by the transport.

Only after successful dispatch does the optimistic window begin.

During the optimistic window:

getValue()
    returns the locally requested value

while:

authoritativeValue()
    returns the owner's last reported state

Owner /state remains authoritative. Optimism is a temporary presentation rule, not ownership transfer.

The default optimistic window is:

5000 ms

Action manifest schema#

Actions publish runtime argument metadata.

Each argument contains:

{
  "name": "seconds",
  "type": "integer",
  "required": true
}

Argument names follow the same one-segment naming rules as Resources and must be unique inside the Action schema.

The schema is self-description first. It is published whether or not runtime payload assertion is enabled.

Action invocation#

An Action is invoked at:

<device>/resource/<name>/invoke

The message is transient.

A RemoteAction can use:

action.invoke(payload);

A successful return means the publish was accepted by the transport. It does not prove the remote Action executed or succeeded.

There is deliberately no standard:

/result
/response
request-id
retained action execution state

in the basic Resource protocol.

Action payload#

The maximum Action payload is:

2048 bytes

An empty Action payload is valid.

This differs from Value state, where empty means tombstone/deletion.

With payload assertion disabled#

The default is:

NM_ENABLE_ACTION_PAYLOAD_ASSERTION 0

The canonical payload std::string is passed to the ManagedAction handler without generic schema validation.

With payload assertion enabled#

When:

NM_ENABLE_ACTION_PAYLOAD_ASSERTION 1

schema-aware Action payloads are checked as JSON objects.

The checker:

  • requires all arguments marked required,
  • checks known argument types,
  • allows optional arguments to be absent,
  • allows unknown extra fields,
  • treats an empty payload as an object with no fields.

That tolerance is intentional: new optional fields can be added without forcing older peers to reject a request.

For a zero-argument Managed Action, the accepted asserted payload forms are an empty payload or a JSON object.

Action argument type checks#

With payload assertion enabled:

boolean
    JSON boolean

integer
    JSON integer

float
    JSON number, including integer JSON values

string
    JSON string

time
    JSON string

colour
    JSON unsigned integer or string

struct
    JSON object or array

ManagedAction result#

A ManagedAction returns:

struct ActionResult
{
    bool success;
    std::string result;
};

Raw MQTT /invoke has nowhere to return that result, so the result is consumed locally and only failures are logged.

Code using ResourcesManager::executeAction() directly can preserve the ActionResult.

The current controlled-console/MQTTP transport is a command request/response mechanism; it is not an automatic Resource /invoke result channel. See MQTTP.

Event#

An Event reports that something happened:

Value  = something is
Action = please do something
Event  = something happened

It is not state. A door that is open is a Value; a button that was pressed, an infrared code that was received, or an acoustic beep that was emitted is an Event. An Event has no current value, so there is nothing to retain, restore, age or read back.

Event publication#

A Managed Event publishes each occurrence at:

<device>/resource/<name>/event

The publication is never retained. A broker therefore never replays an old occurrence to a late subscriber or to a reconnecting consumer, and a Managed Event never re-fires anything on its own reconnect.

The payload is the direct NetCodec<T> encoding of the occurrence's value, exactly as for a Value, with no JSON wrapper, sequence number, timestamp or request id:

bool      true
integer   42
float     23.5
std::string    raw string
custom T  the custom NetCodec<T> encoding

The maximum payload is 2048 bytes. Unlike an Action payload, an Event payload cannot be empty: an empty message is not an occurrence.

Delivery#

Delivery is best effort and transient:

no retain
no replay
no acknowledgement
no deduplication

Two occurrences with equal payloads are two occurrences. fire(42) twice publishes twice, and a consumer's onEvent runs twice. Nothing compares a payload with the previous one, because an Event has no previous value to compare with.

A successful fire() means the transport accepted the publication. It does not mean any consumer received or handled it.

Event ingress#

A Remote Event subscribes to its owner's exact /event topic. For each message:

  1. the payload is decoded through NetCodec<T>,
  2. lastUpdateMs() is set to the local millis(),
  3. onEvent is called.

Every valid message calls onEvent, including consecutive identical payloads. A payload that is empty, oversized or fails to decode is dropped: onEvent is not called and lastUpdateMs() does not change. Like every message about another device, it is still passed to the application's generic NightMare::OnMessage handler afterwards, unchanged.

The transport's retain flag is part of ingress, and the protocol enforces the no-replay rule itself rather than trusting publishers:

/event + retain=false  ->  an occurrence
/event + retain=true   ->  invalid, ignored

A retained /event message (from a buggy client, old firmware or a manual publish) would be replayed by the broker to every new subscription, so delivering it would turn a reconnect into a phantom occurrence. The Resource Manager logs a warning and neither calls onEvent nor changes lastUpdateMs(). The application's generic NightMare::OnMessage handler still receives it with its retained flag set, so it can tell the difference.

A Managed Event never receives /event traffic, so its own publications are not routed back to it.

Event timestamps#

lastUpdateMs() is a local millis() timestamp:

  • on a Managed Event, the latest occurrence the transport accepted,
  • on a Remote Event, the latest valid live occurrence received.

It is 0 until the first occurrence, and 0 again after setSource() or clearSource() changes the source, because occurrences from the previous source no longer describe what the Resource points at. It is not an epoch time and not part of the wire protocol.

What an Event does not have#

An Event deliberately has none of the Value machinery:

retained /state
current value / last payload
availability
freshness / staleness
advertisement period
hardware policy
dependency (dependsOn)
optimistic state
/set write handling
/invoke

The device-wide <device>/status presence says nothing about an Event either. A Remote Event that has not fired is not stale; it simply has not happened.

Resource binding and subscriptions#

Binding a Managed Resource:

  • validates the local device identity and Resource name,
  • locks the device address,
  • adds it to the registry,
  • publishes the updated manifest,
  • publishes state immediately if a Managed Value already has authoritative state,
  • subscribes to /set or /invoke when required.

A Managed Event does none of the last two: it publishes no initial occurrence and subscribes to nothing, because nobody writes to or invokes it.

Binding a configured Remote Resource:

  • validates the remote device and Resource address,
  • rejects a source pointing at the current device,
  • prevents duplicate local representations of the same remote address,
  • subscribes to the remote manifest,
  • subscribes to owner /state for Remote Values,
  • subscribes to owner /event for Remote Events.

A Remote Resource may also be bound before a source is configured. It participates in the local registry but has no network address or subscriptions until setSource() supplies a valid source.

Every Remote Resource has a stable, unique local name. Its source owner and source Resource name are separate routing fields.

Retargeting a Remote Resource#

setSource(device, resource) keeps the Resource role Remote and replaces only its target.

NightMare:

  • unsubscribes old ingress,
  • drops state learned from the old source (for a Remote Event that is lastUpdateMs(); there is no payload to clear),
  • updates manifest subscription ownership,
  • validates the new address,
  • subscribes to the new ingress,
  • stores the binding in /remoteresources.json,
  • republishes the consume manifest.

If the new source is invalid, names the current device, or collides with another bound Resource, the source is refused and the previous source is retained.

Startup restores persisted sources by local name after application binding and before networking. It removes malformed entries and entries for local Remote Resources that no longer exist, but never removes a binding merely because the remote endpoint is offline or absent from a manifest.

Unbinding#

Unbinding a Managed Value removes its retained /state when that Value had authoritative state, then republishes the Managed Resource manifest without it.

Unbinding a Managed Action only requires the manifest update because /invoke is not retained.

Unbinding a Managed Event only requires the manifest update because /event is never retained. There is no tombstone to publish.

Remote unbinding removes subscriptions that are no longer needed, removes the persisted source, and republishes the consume manifest.

Reconnect#

After the active transport reconnects, ResourcesManager rebuilds its exact Resource subscriptions and requests cooperative re-announcement of:

  • the retained Resource manifest,
  • the retained consume manifest,
  • every Managed Value that has authoritative state.

Events are not re-announced. A Managed Event republishes only its manifest entry and fires nothing; a Remote Event rebuilds its /event subscription and does not call onEvent, because nothing arrives that was not published live.

Applications do not need to manually republish all bound Resources after reconnect. tickNightMareESP() processes the manifest, consume manifest, and managed-state requests separately. Failed requests use the capped exponential-backoff policy.

Identity cleanup#

When a device identity is migrated, Resource cleanup under the old device name publishes tombstones for:

  • the old retained state of every currently declared Managed Value.

Actions and Events need no separate cleanup because /invoke and /event are transient.

A Value removed from firmware before cleanup cannot be discovered from the current Resource registry and therefore cannot be automatically tombstoned under the old identity.

Limits#

Current Resource protocol/runtime limits include:

bound Resources per device: 100
Resource segment length:     64 characters
Value/Action/Event payload:  2048 bytes
manifest payload limit:      16384 bytes

Last updated · Edit this page on GitHub