Author Topic: OpenHAB impressions  (Read 7589 times)

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #15 on: January 02, 2015, 05:10:10 PM »
Looks like I broke my session manager at some point, so there are a large number of concurrent sessions. I'll fix that so my poor little Pi can relax a bit. Looking at my session logs, there is quite a bit of action. I'd post a screenshot, but it has IPs in it and I'm too lazy to blur them.

I also dropped Google Analytics code in there, which doesn't help things.

Grabbing 1000 points of data (what most of the channels and IO retain), reliably takes about 4.5s. I could take measures to grab the data in a sparse fashion and bring it in at higher resolution after initial load, but I haven't really futzed with that yet.
« Last Edit: January 02, 2015, 05:21:56 PM by ColinR »
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #16 on: January 02, 2015, 06:03:04 PM »
I think I may give gunicorn a try. I've been meaning to.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #17 on: January 04, 2015, 04:31:27 PM »
I got it up and running with uwsgi and nginx. It's a bit peppier, but configuration is a bit ornery compared with Apache, on the uwsgi side anyway, although there is a great deal of control, albeit with far less community knowledge.

The fact is that there are just bunches of wsgi POST requests going out for data, especially on the controlpanel page, so even at 100ms per complete request it starts to add up. I added in metadata to the returned json that reports how long the db queries themselves are taking, to make sure this wasn't running up against a concurrency wall with sqlite, but this is definitely not the case - although the queries aren't free, so to speak. A typical query that takes 100-175ms consists of <50ms of actual data retrieval and formatting. Another example is the 1000-pt data plot query. It takes 2.8s total, 1.1s of which is actual query + data formatting time. I could further isolate, as the current method of retrieving data is to grab it all and iterate it into dictionaries, which makes is super-easy to work with, and pops right into json for insertion into DOM plot objects.

Once I get my ssl dialed up I'll get this version up.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

kobuki

  • Sr. Member
  • ****
  • Posts: 289
Re: OpenHAB impressions
« Reply #18 on: January 04, 2015, 06:09:32 PM »
Are these response times for an rPi running all components?
« Last Edit: January 04, 2015, 06:56:15 PM by kobuki »

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #19 on: January 04, 2015, 06:48:48 PM »
Yes, it's an RPi running a host of control scripts as a sensor gateway.

The response times are for each POST request from an ajax script to a wsgi application instance. It includes the entirety of sending the request, the server retrieving and formatting the data, and then returning it to the browser. For example, a database query request for a table of channels, or input values, etc. When most pages initially load, content is created dynamically, e.g. a table of channels requires the table be retrieved, and then the table be constructed based on the data. The database is then periodically queried to update data. In the end, I think being creative about avoiding a bunch of simultaneous updates on load will be an important optimization.

You can see all the load operations when you have Firebug installed on firefox in the console window. It will show you all of the ajax requests, headers, and responses in real-time. Beware, however, as according to the docs, having it open slows operations (an interesting conundrum). When I run uwsgi with process/thread monitor open, all requests are essentially as fast as the database queries, meaning it's getting the data in and out very very quickly, but the combination of the web server and the browser processing the data are apparently adding to the overhead.

If you have a critically fast operation, you can just eliminate all other browser operations and it gets quite fast, especially since you don't have to wait for the response for the server to do what you want on the server side. See, for example, here:

http://www.cupidcontrols.com/2014/09/direct-raspberry-pi-io-with-apache-and-wiringpi/

C
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #20 on: January 05, 2015, 04:11:56 PM »
Just ran across a few great ideas on the tubes last night whilst reading about REST philosophy (one of my casual interests).

Anybody hear of ETags? A HTTP1.1 standard header that I'd never heard of. Idea is this:

An ETag is a hash of a previous data response. Each time you sent a specific data request, you send along an ETag that represents a hash of the previous data. It's like a cached copy, but without all the wasted space. When you send along your ETag in the HTTP headers, the server receives it. It compares the ETag with the hash result of the data you requested. If it's the same, it returns a 304 Not Modified instead of a 200 Success, which you can handle in the ajax callback as you see fit. In any case, it will be much faster to receive and handle with no data! Now, typically this is done by attaching an ETag to a specific URI, i.e. GET request, where the parameters specify the exact request you are making. This would mean, however, that POST requests would not cache an ETag properly, with details in the message body. No matter, however, as you can store your own ETag in your js objects that you pass around with your requests if you like, which is what I plan to do.

The advantage of using this instead of something like open sockets or long polling is that you don't need a handler running on the server side to determine when updates have been made and then send them back to the client. Not to say that there is anything wrong with long polling or things like sockets, but I'm lazy and don't want to write server-side code to handle every case of "data has changed, notify client!". The general Not Modified possibility will save overhead on everything with very little work.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

Felix

  • Administrator
  • Hero Member
  • *****
  • Posts: 6279
  • Country: us
    • LowPowerLab
Re: OpenHAB impressions
« Reply #21 on: January 05, 2015, 06:38:47 PM »
I don't see how the client can know that anything has changed and ask for it, other than guessing or polling. It's the server that has to tell the clients that something changed, or an event happened. It's the same as the Moteino end points. The server has no idea when a light goes on. It's the Moteino in the SwitchMote that informs it, and so on.

ColinR

  • Full Member
  • ***
  • Posts: 176
Re: OpenHAB impressions
« Reply #22 on: January 05, 2015, 11:45:12 PM »
I don't see how the client can know that anything has changed and ask for it, other than guessing or polling. It's the server that has to tell the clients that something changed, or an event happened. It's the same as the Moteino end points. The server has no idea when a light goes on. It's the Moteino in the SwitchMote that informs it, and so on.

No, it doesn't /know/ that anything changed. It just periodically requests an update for what data it wants by ajax. The server then responds with either "okay, here's the data" or "nothing has changed since last you asked for that data". The second response contains no data and is immediately discarded, and is therefore much faster.

The problem with server-side notification to the client is that that there is so much data (or at least I have so much data) that you have to figure out a good way for a page to subscribe to changes on certain data, or it would be bombarded by data updates from the server. The other issue, as I mentioned, is that you have to constantly be checking for data changes on the server side. This is not to say it's not a solvable problem. It would just take a think. Something like beginning a long poll with a specification of what data you are interested in would work. You'd have to initialize the data in a previous request though, or somehow convey what the state was last you knew. Anyway, it's been done (Comet) and is what Facebook used to do, along with a bunch of other dudes.
CuPID Controls :: Open Source browser-based sensor and device control
Interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com