RFC: New HiveTool Architecture

10 posts / 0 new
Last post
Paul
Paul's picture
RFC: New HiveTool Architecture

We are entering our sixth year of data collection. It is time to do a complete redesign focusing on self administration, user features, scalability, analytical tools and data warehousing. See this article for more details. Ryan has just released his software suite, HiveControl, that is a complete rewrite of the code that runs on the hive computer with new graphs (based on High Charts) and a bee counter! Ryan has offered to add the new graphs to hivetool.net.

To be fair to Ryan (and anyone else who wants to write code) we need to specify the complete software architecture. This does not need to be extremely detailed, but a general guideline of where we want to go. The current database structure is something I cobbled together and is far from optimal. I'd really like to get input from folks with database design expertise.

If we are talking about redesigning the database for charting, we need to think about other feature we want to add down the road. For instance, we need to capture more meta data, have the hive configuration data flow both ways, and be self administered.

Then as each piece (such as the charting software) gets written there will be less chance that it will have to be rewritten and we stand a better chance of painlessly integrating the pieces into the whole system.

Your ideas, comments, suggestions, criticisms, and insights will be greatly appreciated. Thanks!

Emil
Emil's picture
New Architecture

I have a few things here
- Need to be able to have cheap solar systems which send in data in batches to hivetool.net, normally by 3G/4G networks. The less power comsumption the cheaper equipment can be used.
- Weather information directly to hivetool.net, insted of having to send it to the local hive and then up to hivetool.net
- Alarm sending when different things happens, maybe from hivetool.net?

Paul
Paul's picture
3G/4G (and satellite) low power/compressed data/no local wx

I agree with everything. So, to recap:
1. Hardware and software needs low power/low bandwidth mode. Currently we use xml to upload the data. xml is human readable and we are using long tags like . This could be abbreviated to a few characters or a different format used. Also need to compress it.

2. We definitely want the option for the public server to be able to pull the weather data. The user may still want the hive computer to pull the data if they want to view it on their own box or are running hivetool as a stand alone system. Also, if there are several scales in one location, the weather data only needs to be pulled and stored once.

3. Alarms. This needs to be a separate thread. There are some alarms that have to come from the public server - like loss of signal, due to failure or theft. But, if we have a data acquisition daemon that samples more often (but only uploads in batches), then the alarm would need to originate from the hive computer but could still go through the public server.
satellite

Nate
Nate's picture
Bandwidth ideas

Low bandwidth systems challenges with data collection - The industrial automation world has been dealing with this for years, perhaps take a few ideas that different vendors have implemented over the years. I'll focus on OPC here because it's at least in the US... it's the most commonly used cross platform communication protocol.

User defined scan periods - very low bandwidth systems would greatly benefit from allowing a user to define a slow scan - ie only once an hour. Not all users will be happy with this - users with systems on a good internet connection will continue to want the current or higher fidelity data

Advise vs Polled... Generally there is a greater than 50% reduction or more if only changing data is sent. This is Advise... the client(data reciever) only is advised of a data change. Polled is basically what is used not - Data is scanned at a preset schedule and all data is sent. at that schedule.

Per data stream deadbands of course the ability to choose how much deadband is around every data stream further allows one to fine-tune that data stream and subsequently overall data usage

You could continue developing your custom deamon for handling this, or leverage something like the UPC UA open source that Paul installed on HiveTool for me for both the data server and client. and no I have not had any time to play with actually mapping our data or instantiating a functional data server.

Nate
Nate's picture
Hivetool.net ideas

I'm sure I'll have more ideas
#1 How should hives be organized? I'd propose the following because I would want to see my hives all listed under my username and grouped by Bee Yard
User
Bee Yard
Hive
#2 Where are push notifications done? Hive Controller, Yard Controller, hivetool.net??? multiple engines?
#3 Metadata - is a must to make sense of what's going on - the ability to add notes and put markers from feedings/medications/harvests on a trend the so one can see what preceeds a certain profile
#4 The ability to compare one hive to another. Id like to be able to select two (or if given the option more than two) hives and look at weight or hive temp profiles overlayed

Paul
Paul's picture
sending data/metadata/comparing hives

Again, I agree with everything. A couple of points to consider.

1. Who sends the data. I originally thought that getting one monitoring system per yard would be an achievement. But folks are monitoring several hives per yard (maybe even each hive in the yard). I never considered a yard controller until Ryan brought it up. For simplicity and redundancy, I would probably tend to go with each system being stand alone and sending it's own data up. I would like all the hives to sample at the same time, but randomly stagger the upload times to spread the load out on the server. But I am not against a yard controller, just haven't thought it through. If we go that route, it would be cool if one hive could serve as the controller, but if it failed, it would fail over to another hive that could server as the controller.

2. Definitely need metadata. I never delete bad data or even correct it in the raw database, just flag it as bad. I would like the beekeeper to be able to log in and flag it and annotate it to mark manipulations (like added a super, treated for mites, etc.) Also keep in mind the conversation we had about the maintenance button on the hive.

3. When comparing multiple hives with an overlay, consider "normalizing" the weight so they start at the same point. In addition to comparing hives, I'd like a user on the public server to keep a "favorites" list, so those hive graphs could be displayed on the same page. This would be in addition to seeing just their hives or just the hives in a yard. It would also be useful to average the hives in a yard to get a better idea of what is going on instead of just looking at the strongest or weakest.

Martin
iobroker

Hi guys,
maybe there is also another (different) way to go further.
I am member of a smart home community called iobroker.
http://www.iobroker.net/?lang=en

the architecture is pretty cool:
there is an admin surface and adapters.
The adapter (see link above) are for example
databases (mySQL,...)
a graphic tool (vis) to make own UIs,
interfaces of different protocols (like zWave, philips hue, 1 wire...)
weather services (wu, dwd,...)
push services (pushover)
chart tools (rickshaw, flot...)

more or less everything is already there
everything open source
a powerful community
http://forum.iobroker.net
Bluefox is the main admin and developer
runs very solid on a raspi

from my point of view:
If we would implement our own "Hivetool adapter" (what seems to be not too hard, as many others did it) this could be a solution what brings us in the future.

If you want, contact "Bluefox" dogafox@googlemail.com
to get more information about the architecture and how it works

to try the software on a raspi with this image:
http://www.iobroker.net/?ddownload=2899

Adrian
Roadmap to 2.0 (In the Cloud)

There are some great ideas but I'm hesitant to request new features until we establish which elements need work and in what order. A "complete redesign" is a huge undertaking. Perhaps we need to break it up and focus on the highest priorities and refine and add features once the core is operational.
Let me preface this be constraining my comments to HiveTool 2.0 in the cloud. Yes the hive monitor hardware and software are still in full development, I'll address that separately.
Some of the key elements needing attention;
Hive Monitor Registration & Testing
Data Repository structure
Data Upload & Access
User Interface
Chart Graphics
Personnel - Roles & Responsibilities

Have I missed any?

Adrian Ogden

Emil
Emil's picture
NB-IoT

I'm not a programmer as I have said before, but here must be some extreme ways of getting a system not to use almost any power
Check out the new SARA-N2xxx chip
https://www.u-blox.com/en/blog/narrowband-iot-reaching-devices-other-tec...
and
https://www.u-blox.com/en/product/sara-n2-series
In Norway the NB-IoT network is starting to be installed in 2017, which gets a lot of possibilities in using hivetool in different areas, butmaybe we need to rewrite to ARM design, where I can see that there already is libraries for DHT22, HX711 and TSL2591

Emil
Emil's picture
NB-IoT more info
Log in to post comments