Smart Micro-scale Energy Management and Energy Distribution

In Decentralized Self-Powered Networks Using Multi-Agent Systems

Stefan Bosse
University of Koblenz-Landau, Fac. Computer Science, Germany
10.9.2018

Overview

Introduction

Goal?

Self-organizing energy management and distribution in autonomous and self-powered Sensor Networks, the IoT, and Smart Energy Grids

Methodology?

Mobile Multi-agent Systems that are capable to transfer (virtually carry) energy between networks nodes with Divide-and-Conquer approach

Technology?

Sensor and computer nodes connected by shared data-energy links (i.e., transferring data and energy over a wired or wireless link)

Investigation?

Global emergence behaviour of self-organizing energy management agents

Evaluation?

Simulation of a mesh-grid Sensor Network using the SEJAM2 simulator

Motivation and State of the Art

Smart Energy Grid

  • Tight coupling of Communication and Power Control

figid

Smart Sensor Networks

  • Autonomous sensor nodes, self-powered, mobile, ad-hoc IoT connected

figsmarthome2

Reference Architecture

Network Level

  • This work made no technology or architectural specific assumptions.

Definition 1. (Network)
A network $W$ is a graph $G(N,C)$ consisting of autonomous nodes $n_i \in N$ and edges $c_i \in C$ connecting nodes (communication channels).

  • Simplification: Although arbitrary network topologies are supported, this works assumes three-dimensional mesh-grid networks

    • The network can be irregular (missing nodes) and incomplete (missing links) → Unicast Peer-to-Peer links

  • Each node can be connected with up to six neighbouring nodes:

fignettopo

Network Level

fignet3D

Fig. 1. Three-dimensional mesh grid network

Node Level

  • Each node is some kind of computer (at least a microcontroller) that performs:
    • Data Processing
    • Sensor Processing
    • Energy Management
  • Each node provides the following modules and components:
    • Data Processor P
    • Communication C
    • Set of sensors (Power, Voltage, Light, ..) S and Signal Processor SP
    • Energy Storage ES
    • Energy Harvesting EH
    • Energy Transformer (Sender & Receiver) ET
    • Agent Processing Platform APP
    • A common data base (tuple space) used for agent interaction

Communication Level

Channels. Nodes can exchange data with their neighbours by using some kind of serial communication links.

  • A communication link can be used for the:

    1. Transfer of data and messages;
    2. Transfer of energy.

  • There are different kinds of physical transmission technologies differing in data bandwidth B, latency τ, and energy capability and transmission efficiency ε:

    • Electrical → high efficiency ε1;
    • Optical → efficiency ε2 < ε1; and
    • Radio-wave based → ε3 < ε2 < ε1.

Issue: Each time a network node transfers energy along a path (na,nb) the total transferred energy decreases exponentially (Π ε)

Technological Architecture

figrefarch

Fig. 2. Network topology (left) composed of nodes with sender and receiver blocks (right) used for data and energy transmission between neighbouring node.

Energy Model

The system energy model is introduced for a common understanding and is not used by the agents!

Energy Transfer

  • Nodes can interchange energy via messages.
    • The data processing system can use the communication unit to transfer data (D) and superposed energy (E).

  • Each message sent from node A to node B is an energy transfer $E_{msg}$ consisting of tokens (bits)

  • Sender unit can mix additional energy tokens with message data D (N serial bits) resulting in increased transmitted energy:

\[{E_{msg}} = N{I_D}{\tau _1} + {I_T}{\tau _2},{\text{ with }}{I_T} \gg {I_D} \]
  • Receiver unit stores received energy $E_{msg} \epsilon_{in}$ extracted from each message in local energy storage.

Energy Supply Topologies

  1. Star, Centralized → External Supply
  • Each network node is connected point-to-point with an energy source.
  • Low loss of energy, but not suitable for extended/large networks.
  1. Grid, Centralized → External Supply
  • Each node is supplied with energy from neighbouring nodes.
  • There is one or there are few external supply point(s).
  • High loss of energy: energy is passed along several nodes (total eff. Ω) from source to destination points:
\[\Omega = \prod\limits_{\forall i \in \{ {\text{PATH}}\} } {{\varepsilon _i}} \]
  1. Grid, Decentralized → Self-powered
  • Each node is supplied by local energy source, and
  • additionally? by neighbouring nodes.

Energy Supply Topologies

figenertopo

Fig. 3. Different energy supply approaches and efficiency

System Energy

\[\begin{array}{*{20}{l}} {{E_{node}}(t)}& = &{{e_0}({t_0}) - \sum {{k_{decay}}(t)} } \\ {}&{}&{ - \sum {{k_{comp}}\tau (A{c_i})} } \\ {}&{}&{ - \sum {{k_{create}}A{g_i}(AC)} } \\ {}&{}&{ - \sum {{k_{comm}}{k_{link}}\sigma (ms{g_i})} } \\ {}&{}&{ + \sum {{l_{conv}}{l_{link}}{e_{i,deliver}}} + \sum {{l_{conv}}h{a_i}} } \\ {{E_{net}}(t)}& = &{\sum {{E_{node,i}}(t)} } \end{array} \]
  • $e_0$: initial node energy (deposit)
  • $k_{decay}$: time-dependent energy decay parameter (energy losses in storage)
  • $\tau(Ac_i)$: energy reduction by agent activity $Ac_i$
  • $\sigma(msg)$: size of a communication message
  • $k_{comm}, k_{link}, k_{comp}, k_{create}$: energy consumption parameters for communication and computation
  • $l_{conv}, l_{link}$: energy conversion parameters (efficiency)
  • $e_i, ha_i$: energy delivery and harvesting

Energy Management and Distribution

Node Classification

Very bad node → E < EAlarm

A node with very low energy, with restricted node operation (only agents arriving with energy are processed, only help emergency agents are sent out).

Bad node → EAlarm < E < EThres1

A node with low energy, resulting in a basically normal node operation but with execution limits (number of agents, agent processing time, agent creation, agent migration, agent class restrictions, increased barriers of processing negotiation).

Good node → EThres1 < E < EThres2

A node with normal energy deposit and a normal node operation state with some resource limitations. All agents are processed and the node agent can create any type of energy agents.

Very good node E > EThres2,

A node with very high energy and normal node operation; only a few or no resource limitations. All agents are processed and the node agent will only create distribute energy agents (if behaviour was enabled).

Example of Energy Distribution

  • Two-dimensional mesh grid
  • Energy distribution between nodes under the control of mobile agents performing negotiation with node agents
  • Goal: Decrease number of bad and very bad (non-operational) nodes

figsemexA

Fig. 4. Left: Energy distribution population map without SEM, Right: with SEM (after 10000 simulation steps)

Multi-Agent System

Agents interact with each other by exchanging data base tuples or by creating agents

Divide-and-Conquer

  • There is no centralized instance monitoring energy and performing energy management (distribution)

  • Each network node performs energy management (EM) within a spatially limited region → multiple instances

  • Global EM goals:

    • Stability of system operation
    • No bad nodes
    • High availability

  • Emergence behaviour: Efficient and equalized energy distribution

  • Basic principles to achieve decentralized local EM with a global emergence:

    • Negotiation, Diffusion, Replication, and Inihibition
    • Self-organization
    • Self-connectivity
    • Self-adaptation
    • Self-healing

Energy Transport by Agents

  • Each time an agent that carry energy tokens arrives at a new node, the energy is stored in the node deposit and a virtual energy tuple is produced in the tuple space.
  • If the agent continues traveling it has to consume the energy token again removing energy from the deposit (on sending).

figenergyTransport

Fig. 5. Agent interaction with tuple spaces exchanging virtual energy tokens via tuples

Agent Behaviour and Classes

Node → Stationary Agent

This agent monitors the node state and power history. It has to initiate appropriate actions, i.e., creation of energy request, help, or distribute agents. The node agents has to asses the quality of the SEM locally and can change SEM strategies.

Request → Mobile Point-to-point Agent

This agent requests energy from a specific destination node, returned with a Reply agent. If the destination node cannot deliver energy (bad node), the request agent dies without a reply.

Reply → Mobile Point-to-point Agent

This agent is created by a Request agent, which has reached its destination node. This agent carries energy from one node to another.

Agent Behaviour and Classes

Help → Mobile Region Agent → Reactive

This agent explores a path starting with an initial direction and searches a good node having enough energy to satisfy the energy request from a bad node. This agent resides on the final good node (found by random walk within a region) for a couple of times and creates multiple deliver agents periodically in dependence of the energy state of the current node. If the current node is not suitable anymore, it travels to another good node.

Deliver → Mobile Path Agent → Reactive

This agent carries energy from a good node to a bad node (response to Help agent). Depending on selected sub-behaviour (HELPONWAY), this agent can supply bad nodes first, found on the back path to the original requesting node.

Distribute → Mobile Region Agent → Pro-active!

This agent carries energy from the source node to the neighbourhood and is instantiated on a good node. It explores a path starting with an initial direction and searches a bad nodes to supply them with the energy from the agent virtual energy deposit.

Negotiation & Marking

  • To avoid high density of help and request agents on a specific neighbour node each help and request agent places temporary markings on the node indicating energy demand on this (good or very good) node (aka. synthetic pheromones) by other nodes.

  • These markings are placed in the tuple space of the node and removed after a time-out automatically.

  • If a new help or request agent arrives on a node and this node has a strong marking it will continue traveling (help behaviour) or dies (request behaviour).

  • Each help and request agent negotiates energy demand with the node agent via the tuple space.

Parameter Set

  • Behaviour of energy agent is parameterized:
\[\begin{mdmathpre}%mdk \mathid{parameter}:\{~\\ \mdmathindent{1}\mathid{energy}_{1}:50,~\mathid{energy}_{2}:300,~//~\mathid{operational}~\mathid{energy}~\mathid{range}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Alarm}}:50,~~~//~\mathid{e}~<~\mathid{energy}_{\mathid{Alarm}}:~\mathid{very}~\mathid{bad}~\mathid{Node}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Thres}_{0}}:100,~//~\mathid{e}~<~\mathid{energy}_{\mathid{Thres}_{0}}:~\mathid{bad}~\mathid{node}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Thres}_{1}}:200,~//~\mathid{e}~>~\mathid{energy}_{\mathid{Thre}_{1}}:~\mathid{very}~\mathid{good}~\mathid{Node}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Deposit}}:50,~//~\mathid{reservoir}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Request}}:50,~~~~//~\mathid{def}.~\mathid{energy}~\mathid{to}~\mathid{be}~\mathid{requested}\\ \mdmathindent{1}\mathid{energy}_{\mathid{Distribute}}:20,~//~\mathid{def}.~\mathid{energy}~\mathid{to}~\mathid{be}~\mathid{distributed}\\ \mdmathindent{1}\mathid{explorationRange}:4,\\ \mdmathindent{1}\mathid{Life}_{\mathid{max}}:~4,~\mathid{Hops}_{\mathid{max}}:~8,\\ \mdmathindent{1}\mathid{inhibitTime}:~20,~//~\mathid{inhibit}~\mathid{request}/\mathid{help}~\mathid{agents}\\ \mdmathindent{1}\mathid{energy}_{\mathid{K}}:~0.95,~~~//~\mathid{energy}~\mathid{conversing}~\mathid{efficiency}\\ \mdmathindent{1}\mathid{energy}_{\mathid{CommK}}:~0.8,~//~\mathid{energy}~\mathid{transfer}~\mathid{efficiency}\\ \mdmathindent{1}\mathid{sem}:~['\mathid{help}'],~//~\mathid{energy}~\mathid{management}~\mathid{strategy}\\ \} \end{mdmathpre}%mdk \]

Algorithm

Algorithm 1. (Energy Management)

\[\begin{mdmathpre}%mdk \mathkw{agent}~\mathid{energy}~(\mathid{parameter})~=\\ \\ \mdmathindent{2}\mathkw{if}~\mathid{energy}_{\mathid{node}}~<~~\mathid{energy}_{\mathid{Thres}_{0}}~\mathkw{then}~\\ \mathid{A}:~~\mathkw{choose}~\mathid{neighbour}~\mathid{node}~\mathid{randomly}~\mathkw{goto}~\mathid{node}~\mathid{and}~\mathkw{request}~\mathid{energy}~\mathid{token}\\ \mdmathindent{5}\mathkw{if}~\mathid{neighbour}~\mathid{node}~\mathid{cannot}~\mathid{deliver}~\mathid{energy}~\mathkw{then}~\\ \mdmathindent{7}\mathkw{choose}~\mathid{next}~\mathid{node}~\mathid{randomly}~\mathid{by}~\mathid{directed}~\mathid{diffusion}\\ \mdmathindent{7}\mathkw{repeat}~(\mathid{A})~\mathid{until}~\mathid{a}~\mathid{region}~\mathid{boundary}~\mathid{is}~\mathid{reached},~\mathid{or}\\ \mdmathindent{18}\mathid{a}~\mathid{good}~\mathid{or}~\mathid{very}~\mathid{good}~\mathid{node}~\mathid{was}~\mathid{found}.\\ \mdmathindent{5}\mathkw{else}\\ \mathid{B}:~~~~~~\mathkw{send}~\mathid{and}~\mathkw{deliver}~\mathid{energy}~\mathid{tokens}~\mathid{repeatedly}~\mathid{back}~\mathid{to}~\mathid{source}~\mathid{node}\\ \mdmathindent{7}\mathkw{sleep}~;~\mathkw{repeat}~(\mathid{B})~\mathid{until}~\mathid{this}~\mathid{node}~\mathid{lack}~\mathid{of}~\mathid{energy}~\mathid{or}~\mathid{end}\-\mathid{of}\-\mathid{life}\\ \mdmathindent{5}\mathkw{end}\\ \mdmathindent{2}\mathkw{elseif}~\mathid{energy}_{\mathid{node}}~>~\mathid{energy}_{\mathid{Thres}_{1}}~\mathkw{then}\\ \mathid{C}:~~\mathkw{send}~\mathid{and}~\mathkw{deliver}~\mathid{energy}~\mathid{tokens}~\mathid{to}~\mathid{randomly}~\mathid{chosen}~\mathid{neighboring}~\mathid{nodes}~\mathid{within}~\mathid{a}~\mathid{region}~\\ \mdmathindent{5}\mathkw{sleep}~;~\mathkw{repeat}~(\mathid{C})~\mathid{until}~\mathid{energy}_{\mathid{node}}~<~\mathid{energy}_{\mathid{Thres}_{1}}~\mathid{or}~\mathid{end}\-\mathid{of}\-\mathid{life}~\\ \mdmathindent{2}\mathkw{end}\\ \\ \mdmathindent{2}\mathkw{if}~\mathid{help}\-\mathid{on}\-\mathid{way}~\mathkw{then}~\mathid{charge}~\mathid{each}~\mathid{bad}~\mathid{node}~\mathid{along}~\mathid{path}~\mathkw{end}\\ \mathkw{end}~ \end{mdmathpre}%mdk \]

Algorithm

Utility Function

  • A utility function evaluates the effect of actions of an agent or a set of agents performed in the past on the environment and with respect to the goals of the MAS

  • The utility function u(run) ∈ [0,1] used by the energy agents evaluates the efficiency of the energy distribution (action sequence run={ai}) and influence the sub-behaviour selection and parameter settings

\[u(run) = \frac{{\sum {{e_{delivered}}} - \sum {{e_{consumed}}} }}{{\sum {{e_{collected}}} }} \]

Agent Platform(s)

All Agent Processing Platforms (APP) used in this work base on the same Activity-based Agent Programming Language Model (AAPL)

AAPL

  • Agent Behaviour: Activity-Transition Graph (ATG)

    • ATG can be modified by agents

  • Activities: Set of actions representing sub-goals

  • Actions: Computation, Communication, Replication, Mobility, Reconfiguration

  • Communication: Tuple spaces, Signals

Overview

  • All AAPL-based platforms support mobile agents (representing mobile processes), agent reconfiguration (core morphing), and networking

figapps

Fig. 6. Different agent processing platforms matching different layers: (Horizontal) Implementation (Vertical) Network domain and host platform

JAM

JAM: JavaScript Agent Machine

  • JAM is implemented entirely in JavaScript
  • Agents are implemented entirely in JavaScript (AgentJS)
  • JAM can be executed on any JS engine (node.js, jxcore, jerryscript, spidermonkey, Browser,..)

figjam
  • Tuple space is hierarchically organized (arity → type signature → key hashtag)
  • Pattern matching of tuples in logn time
  • Agent execution in sandbox environment
  • Security: Different agent roles (privileges); capability protection

JAM

Definition 2. (AgentJS Class Template)

\[\begin{mdmathpre}%mdk \mathkw{function}~\mathid{ac}~(\mathid{p}_{1},\mathid{p}_{2},..)~\{\\ \mdmathindent{2}\mathid{this}.\mathkw{v}~=~\epsilon\\ \mdmathindent{2}\mathid{this}.\mathkw{act}~=~\{\\ \mdmathindent{4}\mathid{a}_{1}:~\mathid{function}~()~\{~..~\},\\ \mdmathindent{4}\mathid{a}_{2}:~\mathid{function}~()~\{~..~\},~..\\ \mdmathindent{2}\}\\ \mdmathindent{2}\mathid{this}.\mathkw{trans}~=~\{\\ \mdmathindent{4}\mathid{a}_{1}:~\mathid{a}_{2},\\ \mdmathindent{4}\mathid{a}_{2}:~\mathid{function}~()~\{~\mathid{return}~\epsilon \},~..\\ \mdmathindent{2}\}\\ \mdmathindent{2}\mathid{this}.\mathkw{on}~=~\{~..~\}\\ \mdmathindent{2}\mathid{this}.\mathkw{next}~=~\mathid{a}_{1};\\ \} \end{mdmathpre}%mdk \]

LUAM

LUAM: Lua Agent Machine

  • LUAM is implemented entirely in Lua programming language
  • Agents are implemented entirely in Lua (AgentLua)
  • LUAM can be executed on lua (pure byte-code machine) and luajit (hybrid byte-code and just-in-time native code compilation)

figlam
  • Tuple space is hierarchically organized (arity → type signature → key hashtag)
  • Pattern matching of tuples in logn time
  • Agent execution in sandbox environment
  • Security: Different agent roles (privileges); capability protecttion

LUAM

Definition 3. (AgentLua Class Template)

\[\begin{mdmathpre}%mdk \mathid{ac}~=~\mathid{class}()\\ \mathkw{function}~\mathid{ac}:\mathid{init}~(\mathid{p}_{1},\mathid{p}_{2},..)~\\ \mdmathindent{2}\mathid{self}.\mathkw{v}~=~\epsilon\\ \mdmathindent{2}\mathid{self}.\mathkw{act}~=~\{\\ \mdmathindent{4}\mathid{a}_{1}~=~\mathid{function}~()~..~~\mathid{end},\\ \mdmathindent{4}\mathid{a}_{2}~=~\mathid{function}~()~..~~\mathid{end},~..\\ \mdmathindent{2}\}\\ \mdmathindent{2}\mathid{self}.\mathkw{trans}~=~\{\\ \mdmathindent{4}\mathid{a}_{1}~=~\mathid{a}_{2},\\ \mdmathindent{4}\mathid{a}_{2}~=~\mathid{function}~()~\mathid{return}~\epsilon \mathid{end},~..\\ \mdmathindent{2}\}\\ \mdmathindent{2}\mathid{self}.\mathkw{on}~=~\{~..~\}\\ \mdmathindent{2}\mathid{self}.\mathkw{next}~=~\mathid{a}_{1};\\ \mathid{end} \end{mdmathpre}%mdk \]

AFVM

AFVM: Agent FORTH Virtual Machine

  • AFVM is implemented either in Software (C,JS,Java,..) or in Hardware (FPGA/RTL)
  • AFVM is a multi-stack multi-processor executing FORTH code
  • Agents are implemented entirely in FORTH and organized in code frames (AgentFORTH)

figafvm
  • Token-based pipelined Multiprocessing
  • Code frames holding entire agent code and state (data)
  • Agent execution in strict sandbox environment controlled by processor
  • Agent processors share tuple spaces and agent management data base

AFVM

Definition 4. (AgentFORTH Code Frame Template)

\[\begin{mdmathpre}%mdk \mathkw{par}~\mathid{p}_{1}~\mathid{integer}\\ \mathkw{par}~\mathid{p}_{2}~\mathid{integer}\\ ..\\ \mathkw{var}~\mathid{v}~\mathid{integer}\\ ..\\ :*~\mathid{a}_{1}~..~;\\ :*~\mathid{a}_{2}~..~;\\ ..\\ :\mathid{f}~..~;\\ :\%\mathkw{trans}\\ \mdmathindent{2}|\mathid{a}_{1}~..~.\\ \mdmathindent{2}|\mathid{a}_{2}~..~.\\ \mdmathindent{2}..\\ ;\\ \mathkw{trans} \end{mdmathpre}%mdk \]

Simulation and Evaluation

SEJAM2

SEJAM: Simulation Environment for JAM

  • SEJAM is a MAS simulation environment on top of JAM
  • SEJAM provides a GUI simulation control and a graphical simulation world
  • JAM agents are extended by a visual property
  • Simulations can be modeled with a JSON+ (extended code version) file
  • SEJAM can be connected to real JAM networks (hardware-in-the-loop simulation)

figsejam2

Live

  • Network: Three layers of 8x5 network nodes (3D mesh grid)
  • Initial situation: 30% (very) bad nodes
  • Energy agents are distributing energy between nodes
  • Finally no bad nodes remain

figsimulation3D

Request Behaviour

  • Reactive request behaviour delivers overall good results

figresultsSemRequestQ2963l[120 nodes total]

Help Behaviour

  • Help behaviour delivers overall best results (fast convergence)

figresultsSemHelp2779[120 nodes total]

Help-on-way Behaviour

  • Help-on-way behaviour can create very bad nodes (non-operational)

figresultsSemHelponway1458[120 nodes total]

Event-based Behaviour

The entire MAS is self-organized and self-managing based on:

  • energy perception,
  • reactivity → event-based,
  • inhibition,
  • replication,
  • diffusion, and
  • negotiation.

figresultsSemRequestAgents

Conclusion

  1. Decentralized goal-driven energy distribution was performed with mobile Multi-agent Systems

    • Agent are able to carry (virtual) energy tokens
    • Energy can be exchanged between nodes based on agent negotiations
    • Local behaviour ⇒ Global Emergence (E.g., goal to minimize bad nodes)

  2. No particular system, energy, or network models are required → model-free approach

  3. Agents were implemented in this work in JavaScript and executed by the JavaScript Agent Machine (JAM)

    • JAM can be executed on a wide range of devices and host platforms (embedded computer .. server)

  4. One major field of application (but not limited to): Dynamic Sensor Networks with ad-hoc connected and self-powered autonomous nodes