Author Topic: OpenHAB impressions  (Read 7832 times)

davinci

  • NewMember
  • *
  • Posts: 12
OpenHAB impressions
« on: December 18, 2014, 04:45:03 PM »
EDIT: split topic by Felix(admin): This thread was split from the original thread here to keep them more relevant.

I've looked at your website ColinR, excellent work and design :-)

Thanks for the information. Looking forward to read about the lab you set up for your friend.

The more I think about the nodes and potential relationship with other systems (lights, sonos) in my house I'm probably going to install and configure OpenHAB, evaluate the functionality and see if it will provide the functionality and flexibility I am looking for, and hopefully it will be stable enough on a Raspberry Pi.

I do not want to fiddle and code too much for these solutions at the moment. After the evaluation of OpenHAB i will probably quickly figure out if it will be as expected or not, and then continue with OpenHAB, or go either your way with a python based gateway or a nodejs one.

OpenHAB also got a web ui that is based on a javascript framework, it looks promising: https://github.com/openhab/openhab/wiki/TouchUI

Also like that the OpenHAB system have this rules based trigger system built-in, making it very easy to perform (script/code based) actions based on values from any of the nodes/inputs.

-Davinci
« Last Edit: December 20, 2014, 10:33:00 PM by Felix »

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #1 on: December 20, 2014, 03:17:14 AM »

OpenHAB also got a web ui that is based on a javascript framework, it looks promising: https://github.com/openhab/openhab/wiki/TouchUI

Also like that the OpenHAB system have this rules based trigger system built-in, making it very easy to perform (script/code based) actions based on values from any of the nodes/inputs.

-Davinci

Thanks for the kind words.

Mine is jQuery and jquerymobile (javascript) based. It supports actions out of the box. I just deployed a laboratory pressure sensor Moteino system that emails when pressure drops below a specified level. Took no special code except addition of actions through the normal UI. Easy stuff.

I'll post soon.

C
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #2 on: December 20, 2014, 03:22:33 AM »
Man, I just checked out (or tried to check out) the framework you linked, and:

1. No demo. links broken.
2. No docs.
3. Runs on non-standard ports (therefore not on internet without forwarding or other port-fu)
4. Runs on websockets and uses longpolling.

Mine runs on 100% apache and sqlite. A production web server, not some half-baked dev server they cooked into a "REST API".

Sorry to sound pessimistic, but I'm getting tired of poorly built crap.

C
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6499
  • Country: us
    • LowPowerLab
Re: OpenHAB impressions
« Reply #3 on: December 20, 2014, 01:18:57 PM »
Man, I just checked out (or tried to check out) the framework you linked, and:

1. No demo. links broken.
2. No docs.
3. Runs on non-standard ports (therefore not on internet without forwarding or other port-fu)
4. Runs on websockets and uses longpolling.

Mine runs on 100% apache and sqlite. A production web server, not some half-baked dev server they cooked into a "REST API".

Sorry to sound pessimistic, but I'm getting tired of poorly built crap.

C

Were you referring to OpenHAB?
I looked into that as well, too bad it's built on JAVA, I tried to read on the details but there's not a lot and I don't like the idea of custom ports like 8080/8443 and still not sure what webserver they use.
I am working on a new interface for my stuff which I will make available after I have a working solution, but I'm looking forward to see what you have to share. I also use jquery mobile and I use the no frills standalone neDB database, so far I'm liking the ease of use.

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: OpenHAB impressions
« Reply #4 on: December 20, 2014, 01:32:13 PM »
Sorry to bark in, guys, OpenHAB seems to be popular these days and I'm looking for a similar solution. What is the problem with Java or the non-standard ports in your opinion? 8080 and 8443 are very common port alternatives for the HTTP and HTTPS protocols, respectively. They are mostly in use to be able to run more than a single HTTP(S) service on a single host, on a single IP address and they're configurable. And, since it's written in Java, probably has a built-in web server written in Java, too (most likely candidates are Tomcat and Jetty, haven't read up on the specifics yet). But I'm eager to see your own solution as well :)

(Disclaimer: I have 15+ years of Java SW engineering experience.)

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #5 on: December 20, 2014, 03:53:05 PM »
Man, I just checked out (or tried to check out) the framework you linked, and:

1. No demo. links broken.
2. No docs.
3. Runs on non-standard ports (therefore not on internet without forwarding or other port-fu)
4. Runs on websockets and uses longpolling.

Mine runs on 100% apache and sqlite. A production web server, not some half-baked dev server they cooked into a "REST API".

Sorry to sound pessimistic, but I'm getting tired of poorly built crap.

C

Were you referring to OpenHAB?
I looked into that as well, too bad it's built on JAVA, I tried to read on the details but there's not a lot and I don't like the idea of custom ports like 8080/8443 and still not sure what webserver they use.
I am working on a new interface for my stuff which I will make available after I have a working solution, but I'm looking forward to see what you have to share. I also use jquery mobile and I use the no frills standalone neDB database, so far I'm liking the ease of use.

First of all, let me apologize for my rant, which was fueled by early morning grumpiness and quite a few pints. For the record, I've poorly made plenty of crap in my time.

I was trying to check out their visualization, TouchUI or whatever. It was not meant to be a commentary on OpenHAB by any means. Not that popularity necessarily indicates quality, but they seem to be doing alright. As you mentioned, however, JAVA (boo for me).

I had a demo online of my UI that's connected to my home gateway, but it seems I neglected to update my DNS when my ISP recently switched my IP. Let me get on that. The URL is cupidcontrol.com, so when it flips over you'll be able to take a look. You can use the user/pw viewer/viewer. What I use almost exclusively is the mobile interface (click Go to Mobile at right). I haven't removed the original CSS interface because I still use one of the screens for running my reflow oven. Once I convert it I'll remove it. Many of the old pages already don't work as I have made many js library changes and not maintained them.

C
« Last Edit: December 20, 2014, 04:20:42 PM by ColinR »
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #6 on: December 20, 2014, 03:59:18 PM »
Sorry to bark in, guys, OpenHAB seems to be popular these days and I'm looking for a similar solution. What is the problem with Java or the non-standard ports in your opinion? 8080 and 8443 are very common port alternatives for the HTTP and HTTPS protocols, respectively. They are mostly in use to be able to run more than a single HTTP(S) service on a single host, on a single IP address and they're configurable. And, since it's written in Java, probably has a built-in web server written in Java, too (most likely candidates are Tomcat and Jetty, haven't read up on the specifics yet). But I'm eager to see your own solution as well :)

(Disclaimer: I have 15+ years of Java SW engineering experience.)

Nothing wrong with Java (if you write Java).

Nothing wrong with non-standard ports, if you don't mind typing them into the address bar, and making sure your firewall(s) between your device and the tubes are configured properly. But the better question: why non-standard ports at all?

I assume that you are correct that they run a robust Java web server. My commentary was more general, with reference to application service packages that ship with development servers that people just end up using in deployment. They're just not designed for it, but it's done anyway. Otherwise, you end up doing work to squeeze this special service (that supposedly makes your life SO much easier) through Apache by, for example, by setting it up as a wsgi handler. Which then begs the question: why not just use plain wsgi in the first place? If it's good enough for Django, it's good enough for me.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: OpenHAB impressions
« Reply #7 on: December 20, 2014, 04:29:03 PM »
You're right regarding the inconvenience associated with the nonstandard ports. As for why to use them I think I already replied to that. And while it's surely another small inconvenience for the average user, they can be set to the standard ones. Or you can reverse proxy them if you want.

Application service packages that ship with deveopment servers -- it's a possibility, but servers used for development are usually the same as the ones used for production (in case of Java embedded/bundled servers, at least). The configuration is tailored to suit the usage pattern. That's the issue that is sometimes forgotten, but very important.

As for the choice of technology -- the devs are free to choose any technology they want and so you are.

We're getting a little offtopic :-X

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #8 on: December 20, 2014, 04:47:43 PM »
You're right regarding the inconvenience associated with the nonstandard ports. As for why to use them I think I already replied to that.
Yes, but can you name a use-case where you need to serve two services on the same host where it doesn't make sense to simply separate services by URI within, say, an Apache Directory Directive?

Using another port for another web service has other disadvantages as well. You can't easily call resources on that separate web service from another without explicitly (and annoyingly) configuring Cross Origin Resource Sharing, as calling on another port is viewed as a legitimate security concern. It just seems to make more sense to do the interpretation within the single service (Apache, for example), separating the calls by URI rather than by what port you are calling on.
Quote
Application service packages that ship with deveopment servers -- it's a possibility, but servers used for development are usually the same as the ones used for production (in case of Java embedded/bundled servers, at least). The configuration is tailored to suit the usage pattern. That's the issue that is sometimes forgotten, but very important.
Flask and webiopi are two that come to mind. Flask is eventually run through wsgi, and webiopi is an API that from what I can tell needs yet another application to squirt it onto the net, and is doing so using their custom app server. Why do this when wsgi/ajax/apache already exist, are bulletproof, widely adopted, and well-tested? It's just unnecessary complication for a non-production solution.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: OpenHAB impressions
« Reply #9 on: December 20, 2014, 05:14:49 PM »
Yes, I know a lot of practical scenarios. One of the more common ones is a shared hosting site with eg. Apache as frontend and reverse proxy for a J2EE/Java Servlet-based service running on the same host, like a Wiki, a CMS or a Lucene/SOLR search engine. From the end user it's all hidden most of the time, naturally (and this is important!). The security issue you mentioned is a valid one but needs to be addressed separately. There's a LOT more than just Apache vhosts, WSGI, PHP or Python, etc. Or I could write about techniques for saving public ip addresses by shifting/modifying standard ports and stuffing them behind a reverse proxy, but this is a different story. And so on.

And in reality, there's nothing that's bullet-proof, believe me :) The rest I don't argue with, if you're familiar with the tech and are satisfied with it, use it. Some stuff is ubiquitous and unavoidable anyway, like HTML, JS or AJAX. That's the mainstream.
« Last Edit: December 20, 2014, 05:19:29 PM by kobuki »

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #10 on: December 21, 2014, 08:13:53 AM »
Those are complex applications indeed! I have done a lot of stuff, including reverse proxy (how I get my Pis outside), running a handful of sites at once, all with Apache vhosts and other simple directives.

I finally got around to posting this thermostat project that uses the UI we were discussing:
https://lowpowerlab.com/forum/index.php/topic,802.0.html

I've got a couple more I need to write up as well, but it just takes time.

C
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

davinci

  • NewMember
  • *
  • Posts: 12
Re: OpenHAB impressions
« Reply #11 on: December 23, 2014, 04:05:24 AM »
I'm not experienced with Jetty, the web server OpenHAB is running.

It could be useful to read a little about it if you are unaware of its architecture and history.

This might be an old article, but it's interesting for some.
http://www.infoq.com/news/2009/08/google-chose-jetty

And a list of services utilising Jetty:
http://eclipse.org/jetty/powered/

-Davinci


ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #12 on: January 02, 2015, 01:51:55 AM »
Finally got around to updating a few things and locking down a couple security issues. Still actively developing and tightening things up, so if you get a bug here and there, don't be surprised.

Description here (needs updating to add features recently added):
http://interfaceinnovations.org/ccsoftware.html

Live demo here. User/pw = viewer/viewer . Limited access, no write permissions. Original site is at cupidcontrol.com, but the mobile site will be the only one once I migrate all content.
https://cupidcontrol.com/mobile/

If you get the 'too many active sessions' message, it's because I have a session manager enabled that will limit the number of active sessions per user to a user-defined count. This prevents gateway overload. Each session currently lasts ten minutes (also configurable).

I'll make another dedicated post once I'm sure I've squashed all the major bugs.
C

CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6499
  • Country: us
    • LowPowerLab
Re: OpenHAB impressions
« Reply #13 on: January 02, 2015, 08:50:09 AM »
Wow really impressive. I like the graph integration although it seems a little slow or maybe I'm doing things too fast in the UI and not letting it refresh properly.
Quite a lot of work in there I can tell that.
I see you're using the original mark, have you trademarked your CuPID line of products?

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #14 on: January 02, 2015, 03:56:46 PM »
Thanks. We seem to be experiencing some load issues. Firebug tells me so. Working on it. From what I can tell, my wsgi doesn't seem to be running in daemon mode, which I meant to check on.

Loading the 1000 points normally takes a few seconds, but the other UI stuff is normally very responsive.

Yes, CuPID is registered.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #15 on: January 02, 2015, 05:10:10 PM »
Looks like I broke my session manager at some point, so there are a large number of concurrent sessions. I'll fix that so my poor little Pi can relax a bit. Looking at my session logs, there is quite a bit of action. I'd post a screenshot, but it has IPs in it and I'm too lazy to blur them.

I also dropped Google Analytics code in there, which doesn't help things.

Grabbing 1000 points of data (what most of the channels and IO retain), reliably takes about 4.5s. I could take measures to grab the data in a sparse fashion and bring it in at higher resolution after initial load, but I haven't really futzed with that yet.
« Last Edit: January 02, 2015, 05:21:56 PM by ColinR »
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #16 on: January 02, 2015, 06:03:04 PM »
I think I may give gunicorn a try. I've been meaning to.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #17 on: January 04, 2015, 04:31:27 PM »
I got it up and running with uwsgi and nginx. It's a bit peppier, but configuration is a bit ornery compared with Apache, on the uwsgi side anyway, although there is a great deal of control, albeit with far less community knowledge.

The fact is that there are just bunches of wsgi POST requests going out for data, especially on the controlpanel page, so even at 100ms per complete request it starts to add up. I added in metadata to the returned json that reports how long the db queries themselves are taking, to make sure this wasn't running up against a concurrency wall with sqlite, but this is definitely not the case - although the queries aren't free, so to speak. A typical query that takes 100-175ms consists of <50ms of actual data retrieval and formatting. Another example is the 1000-pt data plot query. It takes 2.8s total, 1.1s of which is actual query + data formatting time. I could further isolate, as the current method of retrieving data is to grab it all and iterate it into dictionaries, which makes is super-easy to work with, and pops right into json for insertion into DOM plot objects.

Once I get my ssl dialed up I'll get this version up.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: OpenHAB impressions
« Reply #18 on: January 04, 2015, 06:09:32 PM »
Are these response times for an rPi running all components?
« Last Edit: January 04, 2015, 06:56:15 PM by kobuki »

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #19 on: January 04, 2015, 06:48:48 PM »
Yes, it's an RPi running a host of control scripts as a sensor gateway.

The response times are for each POST request from an ajax script to a wsgi application instance. It includes the entirety of sending the request, the server retrieving and formatting the data, and then returning it to the browser. For example, a database query request for a table of channels, or input values, etc. When most pages initially load, content is created dynamically, e.g. a table of channels requires the table be retrieved, and then the table be constructed based on the data. The database is then periodically queried to update data. In the end, I think being creative about avoiding a bunch of simultaneous updates on load will be an important optimization.

You can see all the load operations when you have Firebug installed on firefox in the console window. It will show you all of the ajax requests, headers, and responses in real-time. Beware, however, as according to the docs, having it open slows operations (an interesting conundrum). When I run uwsgi with process/thread monitor open, all requests are essentially as fast as the database queries, meaning it's getting the data in and out very very quickly, but the combination of the web server and the browser processing the data are apparently adding to the overhead.

If you have a critically fast operation, you can just eliminate all other browser operations and it gets quite fast, especially since you don't have to wait for the response for the server to do what you want on the server side. See, for example, here:

http://www.cupidcontrols.com/2014/09/direct-raspberry-pi-io-with-apache-and-wiringpi/

C
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #20 on: January 05, 2015, 04:11:56 PM »
Just ran across a few great ideas on the tubes last night whilst reading about REST philosophy (one of my casual interests).

Anybody hear of ETags? A HTTP1.1 standard header that I'd never heard of. Idea is this:

An ETag is a hash of a previous data response. Each time you sent a specific data request, you send along an ETag that represents a hash of the previous data. It's like a cached copy, but without all the wasted space. When you send along your ETag in the HTTP headers, the server receives it. It compares the ETag with the hash result of the data you requested. If it's the same, it returns a 304 Not Modified instead of a 200 Success, which you can handle in the ajax callback as you see fit. In any case, it will be much faster to receive and handle with no data! Now, typically this is done by attaching an ETag to a specific URI, i.e. GET request, where the parameters specify the exact request you are making. This would mean, however, that POST requests would not cache an ETag properly, with details in the message body. No matter, however, as you can store your own ETag in your js objects that you pass around with your requests if you like, which is what I plan to do.

The advantage of using this instead of something like open sockets or long polling is that you don't need a handler running on the server side to determine when updates have been made and then send them back to the client. Not to say that there is anything wrong with long polling or things like sockets, but I'm lazy and don't want to write server-side code to handle every case of "data has changed, notify client!". The general Not Modified possibility will save overhead on everything with very little work.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6499
  • Country: us
    • LowPowerLab
Re: OpenHAB impressions
« Reply #21 on: January 05, 2015, 06:38:47 PM »
I don't see how the client can know that anything has changed and ask for it, other than guessing or polling. It's the server that has to tell the clients that something changed, or an event happened. It's the same as the Moteino end points. The server has no idea when a light goes on. It's the Moteino in the SwitchMote that informs it, and so on.

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #22 on: January 05, 2015, 11:45:12 PM »
I don't see how the client can know that anything has changed and ask for it, other than guessing or polling. It's the server that has to tell the clients that something changed, or an event happened. It's the same as the Moteino end points. The server has no idea when a light goes on. It's the Moteino in the SwitchMote that informs it, and so on.

No, it doesn't /know/ that anything changed. It just periodically requests an update for what data it wants by ajax. The server then responds with either "okay, here's the data" or "nothing has changed since last you asked for that data". The second response contains no data and is immediately discarded, and is therefore much faster.

The problem with server-side notification to the client is that that there is so much data (or at least I have so much data) that you have to figure out a good way for a page to subscribe to changes on certain data, or it would be bombarded by data updates from the server. The other issue, as I mentioned, is that you have to constantly be checking for data changes on the server side. This is not to say it's not a solvable problem. It would just take a think. Something like beginning a long poll with a specification of what data you are interested in would work. You'd have to initialize the data in a previous request though, or somehow convey what the state was last you knew. Anyway, it's been done (Comet) and is what Facebook used to do, along with a bunch of other dudes.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com