Monday, October 5, 2015

An adventure in IoT

Dear diary,

It has been a long time since I last wrote to you ....

Uhm, yeah, whatever....

Lately, I have been too busy at work. Actually, I still am.
Our game went into closed beta and hell broke lose, as usual.
I am not complaining, I like what I do, it just leaves me with no energy to do any of my hobbies when I get home. My favorite joke lately is:

"After a busy day at work, the last thing I want to look at when I get home is code.
I am so happy I am not a gynecologist"

So I had nothing to brag about ... except about work, which I can't do here.

Fortunately ?!?, I got sick these days, which forced me to stay at home for two afternoons and then came the weekend. As much a being sick sucks, it leaves you with plenty of spare time and you better spend it doing something interesting, which keeps you busy, so you don't just lie down and think how you feel like shit all day. If you can.
And, in the mean time, I got an idea for a simple DIY project. Basically, I've always wanted to be able to work from home with my morning coffee (which takes about an hour). Technically, it is not a problem, there are more than enough solutions for remote access, some of them even can handle launching a 3D game and are secure enough. I am using one and it works quite alright. But, there is this minor issue - most computers at the office are automatically shut down every night, because there have been too many cases of power shortages, which led to hardware being blown. Let's not comment how absurd that is in the 21st century, in an area called "Business Park" with 1000s of computers spread in 100s of offices. Anyway, my problem is simple - I cannot remote access my PC while it is shut down.
There are many solutions, like wake up on LAN or simply arrange it so my PC is not shut down (and take the risk that something might blow during the next power spike).
But, I thought of something cooler - a device that would sit on top of my PC's case and physically press its power button when commanded over the internet!
Sounds simple, right? Basically all you need is a little servo and a way to control it over the internet.'
And I have a spare servo lying around - I ordered 3 servos some time ago, 2 of them are used for my birthday present for my cats and the 3rd one was just waiting ... for this moment :)
I also have some ESP8266 boards waiting to be used. If you don't know what that is - it is a little and very cheap device that has WiFi capabilities and can be programmed. It is becoming more and more popular all over the internet. I have written several posts about it: here, here and here.

Oh wait, I forgot my typical disclaimer - this post is going to be full of technical details. It will contain a good amount of ranting, this time towards a group of open source developers you never heard of, developing a product called NodeMCU. If you're not interested in technical details, here is the usual summary: I'm ok, I haven't been blogging lately, because I was (and still am) too busy at work. I finally had some time to tinker with another gadget. I haven't even finished it yet, I don't know if I ever will. But, as they say. what's important is the journey, not the destination (good luck selling that to a publisher, by the way). So, I advise you to continue reading only if you're interested in technical details.

Now, for the (3?) people who made it to here, lets continue :)

At the moment, there are 4 ways to "program" the ESP8266:

1) Using it as a serial-to-WiFi device, by sending modem-like AT commands to its serial port. This requires either plugging it into a (running) PC or using another device, like an arduino. Could work (with an arduino), but I am not a fan of this approach, simply because the ESP itself is programmable and does not need an arduino.

2) Using a leaked SDK - I think no one does that anymore - it is too cryptic and cumbersome. No, thanks.

3) Using esp8266 / Arduino - this lets you program the ESP in the arduino environment in C++, very similar to how you program the arduino itself. This is what I have been using until now to mess around with the ESP. I still think this is the best approach.

4) Using NodeMCU - a firmware which allows you to program the ESP in Lua. I have always been skeptical about that - interpreted language on such a "tight" platform?!? But it is getting more and more popular, and I was feeling adventurous, so I decided to give it a try.

So, I chose NodeMCU this time. The task (just control a servo upon a request from the internet) seemed simple enough. Plus, there is this interesting concept of being able to push software updates over the air - since it is interpreted, you could (theoretically)  send the new code / scripts over the internet and the host code would rewrite its files and reboot. Sounds cool, right? Except that in practice it is nearly impossible. I will probably explain why later.
Also, using method 3 requires you to reflash the ESP every time you upload your program. You need to reset the ESP into programming mode, upload the new firmware (your program) then test. It's easy, but takes much more time than I would like. With Lua, you just upload the new script and that's it (or at least it seemed so in the beginning).

The first thing I tried was to flash an ESP with the NodeMCU firmware, hook up a servo and give it a spin. Worked like a charm, super easy, done in like 15 min. So far so good.

Then I stumbled into the old problem that the ESP does crazy stuff on pins 0 and 2 on boot. I complained about it in previous posts, but basically when the ESP doesn't like it when stuff is connected to its GPIO0 and GPIO2 pins on startup. In the case of a servo attached, the results were frightening - if you boot it with the servo attached the servo begins to produce horrible noises and not move at all. Luckily, it didn't blow up, because I have just 1 spare at the moment. I have dealt with this problem before, but the solution involved too many external components that it just wasn't worth it.
For this project, I just use an ESP-201 - a different variant, which has more GPIO pins broken out. So I leave 0 and 2 alone and use another one (GPIO4 in this case).

Oh, and while I am on the subject of ESP-201 - whoever produced it like this is a fucking idiot. If you don't look carefully at the pictures, it looks like a nice breadboard-friendly board with two rows of 2.54mm spaced male headers and a serial header. On a second look (or in my case when you order 3 and they arrive) you find out that:
- The serial header pins are pointing in the same direction as the other pins. So you can't put it on a breadboard after all. Not the way it is. Some people say they just bent the serial headers so they stay horizontal and then it can be placed at the edge of a breadboard,
- The labels on the pins are on the BOTTOM side! Are you fucking kidding me? How am I supposed to read those labels when I put the thing on a breadboard?
All they had to do was to solder the side headers on the other side, or simply not solder them at all, like most electronics manufacturers do. Now that I think of it, this is the only piece of electronics I have bought that comes with headers soldered and they are soldered on the wrong side!
Luckily, with the use of the right tool, it takes 5-10 min to fix it.

Ok, back to the project.
After I verified that I can control a servo with the ESP, all I had to do is to make it listen for a command over the internet and then move the servo so it pushes the power button on the PC.
And here comes the next problem - I can't simply run a small server on the ESP waiting for commands. Because it has to stay in the office, connected to the office WiFi. But there is no way I can convince the admins to open a port for me (well, maybe there is, but its not worth it). So it won't be reachable from the outside. Hmm.
The obvious solution is to have the ESP connect to some server, which I can reach from anywhere. Like my smart home server. That's a plan!

The simplest way would have been to have the ESP send HTTP requests to poll my home server periodically and program my home server to tell it if it needs to do its thing. But that's lame,
So, I enhanced my home server to also accept plain TCP connections, so the ESP connects once and the connection is kept alive. So now, I send an HTTP request from any web browser (e.g. my phone) from anywhere in the world to my home server, then it immediately sends a message to the ESP (over the persistent TCP connection) to activate the servo. No polling, no lag, good stuff.
I had to refactor my home server, because it was using some pretty basic blocking approach (a web server example I downloaded from the net) which was totally unsuitable for the task. Now it uses asynchronous accepts and receives and can handle multiple connections. Fancy stuff. It turns out simple HTTP was easier than good-old-plain-TCP.
As a side effect, now the two instances of my home server (one in the living room and one on the terrace) are now talking over a single persistent TCP connection, instead of bombarding each other with HTTP requests (asking each other what is the temperature in their room every 5 seconds).
Oh, and I finally found a work around  for a nasty bug in WPF font rendering that caused the clock displayed on the living room PC (running XP) to disappear sometimes.

Now, to the ESP part.
Programming it in Lua turned out to be a nightmare. Something so simple took me so long!
The whole idea of using Lua is to make programming simple stuff a breeze. But not in this case.
I had unexpected problems on every single step. And it is pretty difficult to find out why something does not work when you don't have a debugger, error messages are either non-existent (the thing just reboots without saying why) or obscure, like "attempting to index a nil value", Period. Where? Which value? Which file and line? Sometimes it tells you, but often it doesn't. Add to that that there are bugs in the firmware, so code that should work sometimes just doesn't. Are you starting to feel my pain?

But these are not even the biggest issues.
THE biggest issue is "out of freaking memory". All the time. I am so sick of seeing this message.
Lets look at some specs:
The ESP has a whopping 96KB of data RAM (compared to ~1.5K on the arduino). It also runs at 80MHz (compared to the arduino's 16MHz). So, compared to the arduino, it should be a beast. And it is.
But not if you put a NodeMCU firmware on it and run Lua.
Out of these 96K, after the firmware and runtime, you have around 20K free for your program. The thing is, these 20K get eaten very fast. Your code and your runtime data are all supposed to fin in (remember, in Lua code is data). I had a hard time fitting a simple TCP client (around 200 lines of code with lots of blanks). And that is after you precompile some of the code.

Lets talk about precompiling for a second. Normally, you just write some Lua code, send it to the ESP and it executes it. You can organize it in files saved on the ESP's flash. But once you hit the out of memory message (very soon) you can precompile some of the files, so it runs byte code instead of raw source. It also makes files "load" faster, as if it matters.
But now, the process of uploading a program (split into several source files) starts to become annoying. It looks like this:
- Make changes to the source code
- Upload it to the ESP
- Reboot the ESP (otherwise there would be not enough memory to compile it)
- Tell the ESP to compile it
- Run
If your code is split into several files you have to do that for each one of them.
I am not sure which takes longer - this procedure or just writing C++ and flashing the complete firmware every time.
And still, precompiling helps, but not by much. Right now, my simple client runs with about 7K ram free. If you go bellow 5K bad things start to happen.

Anyway, I pretty much made it. The ESP connects to my server, waits for command and ... blinks an LED when instructed :) I use blinking for testing, in order to complete the project I have to replace the LED blink with the servo code (and hope it fits in memory) and I am done. It has been working rock solid for two days now. There are some occasional drops, but I have programmed it so that it keeps trying, even reboots and keeps trying to connect to my home server. All good.

Then I started thinking how it can be made better. I mean, programming in Lua for the ESP in general, not the current project.
What was bothering me is the way you code for it.
On the NodeMCU homepage they put this in the list of features:

Nodejs style network API
Event-driven API for network applicaitons, which faciliates developers writing code running on a 5mm*5mm sized MCU in Nodejs style. Greatly speed up your IOT application developing process.

Well, I have only heard of "Nodejs" without knowing what it actually is. All I know is that is some javascript thing web developers use for something.
But, now I know what event-driven API means in terms of NodeMCU.
Basically, it means this:
con = net.createConnection(net.TCP, 0);
con:on("connection", function() conn:send("TCP ESP8266") end)
con:connect(port, ip)
tmr.alarm(3, 5000, 1, function() con:send("PING mem=" .. node.heap()) end)

instead of this:
con = connect(ip, port, net.TCP) -- block this "thread" until connect finishes
conn:send("TCP ESP8266")
while true do
    sleep(5000)
    con:send("PING mem=" .. node.heap())
end

Now, that is ridiculous!
The whole point of using Lua is so that you can program the second way.
It may be not clear from these simple examples, but programming with callbacks is probably the worst way of doing things. As soon as things start to get a bit more complicated it becomes exponentially more difficult to write the code as well as to read and understand it (i.e. bug fix it and maintain it). Hell, even debugging with a proper debugger (if you have that luxury) is still hard.
The whole beauty of Lua is that you can turn messy call-back spaghetti code into linear easy to write and understand one.
So why on earth did they go through all this trouble porting Lua to the ESP when they ultimately give you THIS?
If you still have some doubts, try porting the following simple piece of code in the "callback" paradigm:
for i=1,5 do
    print(i)
    sleep(1000)
end
It will probably look something like this:
i = 1
print(i)
timer(1000, function()
    i = i + 1
    if (i > 5) then stop_timer() end
    print(i)
end)
Now, NodeMCU can suck my balls. And, if Nodejs is like this ... well, now I know why I have 2 balls :)

Then I though I should do it myself - not suck my balls - provide linear programming style for NodeMCU. After all, they say coroutines work in their firmware, so it can be done.
Except they don't.
I tried.
In general, they work, but not always.
It turns out, you can't coroutine.resume() in a socket event callback, like on("connect")
So, I googled it and found a bug report issued by some guy complaining about the same thing.
And here is the response by someone nicknamed "TerryE" (I don't know for sure, but I am assuming the guys is one of the devs):

the nodeMCU eLua model is event driven. Use the event system to implement concurrency. Coroutining across event threads won't work.

Dear Mr. TerryE,
Both of my balls are now occupied, so you have to suck my dick!

Or, in a more polite manner:

You asshole,

I don't mind your software having bugs (its open source under development, that's understandable),
but telling people they shouldn't try to use language features that MUST work (if you pretend you support the language) just because it is not the programming style you had in mind - that is unacceptable. Especially if it is the worst programming style possible, but that is a matter of another discussion. For the sake of strict bug reporting - there is a language feature not working properly. Either fix it, or put a big disclaimer on some front page explaining why it does not work and cannot be fixed. Or even better, go kill yourself somewhere quietly and have someone else fix it.

I think I am done now.

Oh, by the way, I found a work around that bug and did implement a coroutine-based linear-style TCP client. The problem is, pretty much the same code now runs with around 3K memory left (due to the extra code providing the framework) and does not run stable (due to the low memory and due to other bugs in the firmware).

Ok, I really am done now.

P.S. The new prototype version of the TCP client code (without the framework) looks like this:
function Client(ip, port)
--on("error", function(thread, event, ...) log("Error", unpack(arg)) end)
while true do
local socket
while true do
log("connecting ...")
socket = connect(ip, port, 5000)
if socket then
log("connected")
break
end
log("failed")
sleep(5000)
end
sleep(1000)
log("logging in ... ")
if send(socket, "TCP test", 5000) then
log("success")
sleep(1000)
while true do
log("sending status ...")
if not send(socket, "PING mem=" .. node.heap()) then
break
end
log("sent")
sleep(1000)
log("requesting temp ...")
if not send(socket, "temp", 5000) then
break
end
log("sent")
log("receiving ...")
local res = receive(socket, 10000)
if not res then
break
end
log("received", res)
while true do
res = receive(socket, 0)
if not res then break end
log("received", res)
end
sleep(5000)
end
end
log("failed")
sleep(5000)
end
end


Thursday, July 30, 2015

Happy Birthday, Leona and Foxy!

Today my cats become two years old!
Happy Birthday!


It has been wonderful two years - they have brought me lots of laughter and lots of comfort.
There is nothing more relaxing than the cat purr - it washes away all anger, all troubling thoughts, all bad mood.

Yes, there is cat hair everywhere. Yes, it is hard to sleep when they decide to play chase or wrestle at 6am or when a 7kg monster walks on top of me and gives me a massage. Yes, there is some minor damage to the furniture and the curtains. Yes, I can't leave any small objects (like screws, cables, electronic components, lego parts, etc.) on the table unattended. But it is totally worth it.

They have changed quite a lot for these two years - they are much bigger now, less playful, way more lazy. But they are just as cuddly and kind as always. And they have changed from extremely cute little (ok, maybe not little) kittens into enormous but gorgeous cats.

Here is a little present I made for them:

It is not final - it needs some tweaking and proper packaging, but it works.

And, here are some pictures taken over the years (in more or less random order):

















Tuesday, July 14, 2015

Cats and math

Two days ago I cut my foot by stepping on a piece of broken glass - obviously Billy did a lousy job cleaning that.
It wasn't anything serious, but enough to leave blood drops on the floor.

Then I looked at Leona:


And I did some quick math:
- The table dimensions are 80cm x 80cm
- She is probably about 1 meter long with her legs extended
- I've seen her jump easily over 1 meter high from standing still
- She is incredibly strong
- She might look fat, but she is still quite agile
- She weights more than 7.5kg

This is Foxy:
She doesn't look that fat, but still she weights over 7kg and is faster than Leona and almost the same size.

And, their claws are very very sharp.

They both are incredibly kind and cuddly, they would never hurt anyone on purpose. 

But ... if they taste blood? The blood I am spilling on the floor?
Home might become a very dangerous place to live.
They could reach my balls without even lifting from the ground and they could easily go for my throat if they jump. Before I can even react.

Boy, was I quick to wipe it from the floor!

:)


Sunday, July 12, 2015

Meet F.A.R.T.

I am proud to present the first prototype of my Fully Autonomous Roaming Tank (F.A.R.T.):







It is a "robot" tank that roams the house by itself without crashing into things (too much).
What is most special about it is that it is built from Lego parts (chassis an motors), but driven by Arduino!
All I had to do was to cut one of the Lego's wires, do some easy soldering and voila - I can drive Lego motors from Arduino!
The possibilities this opens are huge, FART is just the first example.
Both the code and the mechanics are just quick prototypes - about 2 hours to build the chassis from Lego and maybe an hour coding ... no wonder it doesn't behave too well.

Just as a reference, more than an year and a half ago, I built something similar, but made entirely of Lego (using the Lego brick as a brain, not an Arduino):


Here is the parts list:
1x Arduino Nano - $3.20 (less if you buy more at once)
2x US100 Ultrasonic Sensors - $7 (for both)
2x Transparent Breadboards - $5.24 (for both)

Did you spot the biggest problem already? :)
Lego is fucking expensive!
Since I already have it anyway, it is not that much of a problem.
But, the price (along other problems) makes it useful only for prototyping.
I can't build something and keep it like this, because the price of that thing would be insane.
So, the plan is to replace it with this $10 kit when it arrives.
Still, the option to quickly prototype something with Lego is priceless!

On the other hand, if I wanted to buy an ultrasonic sensor for the Lego, I would need to spend a fortune ($60+), instead of $3.50 from China.

Plans for future improvements are:
- Improve the code for better obstacle avoidance, stuck detection and unstuck procedure.
- Make it internet-enabled, so it can be controlled from any web browser, including from the phone.
- Add some more sensors (maybe 1-2 infrared distance sensors).
- Add some servos, so I can rotate the sensors, like in the pure Lego roamer.
- Add FPV (a camera streaming video of what the robot sees). Ideally, over the internet.
- Replace the breadboards with PCBs
- Replace the Lego with the final chassis

If all goes well (and I don't just abandon the project), at the end I will have a tank, which I can control over the internet and see what it sees. How cool would that be? Well, at least until the batteries die.  

Overall, it was fairly easy to do it, but still I stumbled into few problems:

Problem 1: Ultrasonic sensors take a while to get a distance measurement. And they also need some time between measurements (so they don't pick some echoes). Having 2 of these, I have to measure one first, wait for some time (20ms), measure the second one, wait some more. As a result, I am getting measurements every ~60ms, and the second is about 30ms later than the first. These times might sound small, but actually are big enough to cause problems.

Problem 2: Speed controlling the motors using PWM from the Arduino turned out to be problematic. At low speeds the motors don't turn at all, jut make some noise. I am not sure exactly why, but I suspect the PWM frequency is too high. I could be wrong, but at least I can test that later and if needed fix it (either by reprogramming the Arduino timers or by manually setting the signal low-high from code). For now, I just put a lower limit at the speed control, so if the logic says it wants the motor rotating at 30% speed it actually rotates at 75%. Far from ideal, but enough for the early prototype. It might also help if I use a 9V step up regulator to provide 9V to the motors (currently the batteries provide 7-8). Testing with 9V wall power supply showed it is better, but does not eliminate the problem. I will have the 9V voltage regulator in a few days.

Problem 3: I am running out of digital pins. Motors take 2 pins each. Sensors also take 2 pins each. Adding and ESP8266 for internet connectivity would eat 2 more pins. If I want to read the motor encoders (so I can detect when a motor is stuck or balance the speed of the two motors) it would take 1-2 pins per motor (the encoders output through 2 pins, but I could only use one if I only want to detect if the motor is stuck). Adding extras, like servos, more sensors, speakers, LCD display, LEDs might turn out impossible (without adding second arduino or a GPIO extender).

Problem 4: I somehow burned an Arduino. Fartie was roaming around just fine, then got stuck under a chair and out of a sudden it stopped. It surprised me, because the logic it uses never stops. Then I saw the light I have programmed to blink when the batteries get low was actually blinking. This explained the stop, except the batteries were freshly charged. I measured them just in case - they were indeed fine. But since I unplugged the batteries, the Arduino won't boot from batteries anymore. It boots from USB, but not from batteries. And when hooked up to batteries its 5V output produces something like 1.5V. So, I am guessing its internal 5V regulator is gone. Maybe, because Fartie was stuck, the motors tried to draw too much current (I measured 1.5A, even 2A stall current before). The motor driver chip does have over current protection, but still, something might have happened to the battery power supply, which went through the Vin on the Arduino and fried its voltage regulator. This is only a theory though, I don't know for sure. What is worse, I have no guarantees that it won't happen again and kill the next Arduino.

Problem 5
Cat hair! Lots of it!
There is no solution to this problem (getting rid of the cats is not an option at all).
Billy (my robotic vacuum cleaner) suffers from the same problem.



Tuesday, July 7, 2015

You can stop bracing yourselves now, the money has come.

"Better late than never" they say.
I agree :)

Today I was greeted with this e-mail:





In other words, the first money finally came.
They were supposed to come on June 30st, but "due to internal issues" (according to Samsung themselves) they were delayed 1 week.

Not really a big deal for me, given the fact that my app started selling on April 19th and by the end of April it made $211 (according to their sales statistics). So, whether it is 2 months later or 2 months and one week later - it doesn't make much difference.

Anyway, the drinks are on me tonight :)

On a side note, their Sales Summary seem to be broken since July 1st.
They keep showing just one sale (in Korea) since then.
In the meantime their download statistics show "normal" rate of downloads.

Did you have any trouble understanding the above 3 sentences?
Because Samsung's support obviously did.
I wrote to them yesterday, to notify them of the problem and today they responded, asking for screenshots?!?
Why the fuck do they need screenshots?!? To understand the problem better? Well, I believe it was clearly described in 2-3 short sentences. For proof?  As if I have no better stuff to do, but mess around with them?
I think the answer is different - it is just laziness. The monkey in their support center is trained to ask for screenshots every time an issue is reported and no screenshots are attached. It is so much easier - ask for screenshots, bam! case closed, my job here is done ... next please.

Anyway, I resisted the urge to respond in a ... not so friendly manner ... and very politely sent them these 2 screenshots:





If I didn't already know what their answer will be, I would be really curious :)
As a matter of fact, this is the first time (in my experience) when their accounting data fucks up.
Until now, the statistics were breaking every couple of weeks, but the accounting data looked ok.
But, after so many problems, one more does not surprise me at all.

Ok, enough blogging for today, I have $472 to spend, starting immediately.
Let the shopping spree begin!


Saturday, June 27, 2015

Pimp Your Watch (a.k.a. WatchPlus update 1.04)

I finally did it - took a break from electronics and stuff, sat on my ass and did the update I promised quite a while ago.

Main features of the new update are:
- Added option to use an image from the watch Gallery as a background
- Added 12 color options (there were no color options before)
- Added option to display seconds

In my opinion seconds are totally pointless and changing the foreground color may be cool but makes visibility worse. But, people asked for these, I promised and delivered.
The idea to use Gallery images as a background was really nice one though . BIG thanks to the dude who suggested it. I only have his e-mail though, so I am not shouting it out :)

The outcome is that you can seriously pimp your watch now :)
And, if a picture is worth a thousand words, here are 15K words:






You know what's funny - I am not going to hate Samsung in this post! Maybe just a little.
I expected to face lots of troubles implementing these features, but at the end they turned up fairly straight-forward.

Well, the 1st attempt to get pictures from Gallery failed miserably. I tried reading the files in the images folder, loading the images and showing them to the user to pick from. It worked, but it was horribly slow. That was because it had to stretch many large images at runtime. I didn't know how to pre-scale an image back them (remember, I am still a noob in this web tech thing).
Fortunately, there was an alternative way - pretty much an API call, which invokes the system default's image picker. There even is a sample how to do that in their SDK.
Of course, I still had to scale the resulting image (the background the user has chosen), but I figured out how to do that anyway (more on that later).

The other "challenge" I expected was how to colorize images - some of the clock styles use bitmap images with built-in light effects in them. So, to change their color I had to process the image and colorize its pixels.
I found many examples how to do that on the web (it is just common HTML5 tech, not Samsung-specific). The pleasant surprise was that it just worked (with my experience with Samsung so far I expected something will go wrong along the way).

The process of "processing" an image looks like this:
- create an offscreen canvas
- draw the image in it (scaling is possible)                                                          (1)
- get a copy of the canvas pixels (an array of RGBA values)                                 (2)
- process the pixels in the array any way you want (colorize them in my case)
- "draw" the array of pixels back to the canvas                                                     (3)
- get an "image URL" from the canvas!                                                               (4)
- create a new image, and tell it to load from that URL (as if it loads from file)       (5?)

If you're wondering what these numbers in parenthesis are - I counted the number of (unnecessary) memory copies in the process. Basically you copy the image twice just to get access to its pixels, and then 2-3 times to convert the result back to an image!

The game programmer in me vomited a little.

I'm pretty sure there is an explanation why it must be done like this, but honestly - I don't want to know.

Still, I was surprised that it actually worked ... and not surprised at all that it was slow.
Initially, changing the color of the clock took above 2 seconds! To be honest, it wasn't that bad, just a shame.
I started thinking of performance optimizations, at some point I caught myself thinking whether it will make a difference if I skip pixels with alpha 0 (because they should be already in the CPU cache) ... then I stopped and told myself "Dude, seriously? This is not game dev, not C++ and not PC. This is javascript running god-knows-how (interpreted?) on an exotic platform. Besides, processing the pixels is not what takes the time, the 4-5 memory copies are. Or at least I would guess so without profile data to prove it."
I could profile it with debug prints, I could delay it over several frames, but it just wasn't worth it.
I ended up with some quick-and dirty optimizations which brought it to about a second. Far from ideal, but in reality - who cares!.
At least it is not Samsung to blame here - looks like the WHOLE WEB works like this.

Ultimately, coding the new update took me less than 2 days. Not so bad.

Preparing the update for submission took me another half a day!
Why?
Because I had to prepare new screenshots and the fucking Russian translation!
That Russian translation was a big mistake.
I did it, because Russians are used to see software translated in their language. Several of them asked for translation. So I thought "why not - it would probably increase the sales in Russia".
It didn't.
Instead, sales in Korea grew significantly!
I doubt that is some kind of butterfly effect - more likely Koreans liked the analog clock styles which came with the same update. Or ... my app has been promoted in Korea (I have no way of knowing that) without Samsung telling me.
Anyway, disappointed by our brother Russians, I cut some corners and used the English screenshots in the Russian version (except for the preferences). The difference is only the date anyway.
And this time I translated the texts myself (with the help of google translate, which is crap), instead of bugging my Russian speaking colleagues for help. I don't care.

Finally, all done, checked and double-checked, I go to Samsung's web site and hit the "Update" button. Guess what happened? Nothing. Tried few more times - nope, the "Update" button didn't work!
I laughed out loud.
At the end, logging out and back in to their site solved it, so I was able to submit the update.
If nothing goes wrong, it should be live in about a week.

Thursday, June 25, 2015

Brace yourselves, the Money are coming!

10 days ago, I received this e-mail:



Before I jump on Samsung, let me slap Google first, just for warm-up. I went to my Google mail web UI to get the above screenshot and see when exactly it came, so I could start the 1st sentence in this post accurately. And here is how the line in my Google web UI looked like:


Ok, 9:09 am. ON WHICH DAY?!? Seriously, they tell you the time of the message, but not the day - for the past 10 days or so?!? Messages from June 11th and older have a date instead of time, so it is not like they just forgot to add the date. There is actually a special handling in their code, written by a real programmer and probably designed by a real designer and passed real QA, which makes the timestamps work like this. Fucking amazing!

But, screw Google (for now), this post is all about Samsung.

First, in case you haven't read my previous posts carefully - "Settlement Report" is supposed to come when your profit from selling apps on the Samsung market exceeds $150 and they are about to send you the money.
To cut your enthusiasm early - nope, I still haven't received any money ... yet. But they are coming.
After few days without payment, I decided to write Samsung and ask them when to expect the money. Here is their answer:


• Here is the answer to your enquiry.
Dear Customer,

Thank you for contacting us. We appreciate the opportunity to assist you.

Regarding your inquiry, we would like to inform you that if you have the settlement report confirmed your revenue is exceeded 150 USD, then you may expect the payment to be paid to your PayPal account at the end of the month around 30st. If you do not get the payment by then, please contact us again with screenshots of the settlement report. We will further look into the matter for you.

Besides, please be noted that the settlement cycle begins on 15th of every month.

We hope this information could be useful.

Ok, so June 30th. No rush.

But:

- Why on earth are they sending me an e-mail on June 15th to tell me that they are about to send me some money 2 weeks later?!? What is the point of this report?

- What the fuck do they mean by "If you do not get the payment by then"? Does that happen too? And why am I surprised?

- I fail to explain to myself how "settlement cycle begins on 15th every month" relates to "Period : Apr 19, 2015 ~ May 31, 2015". This is more confusing than women's cycles and periods!

- This one is funny too: "Settlement Status : Settlement Confirm", "Estimated Transfer Payment : USD 472.70". Which one is it - Confirmed or Estimated?!?


And now, let's talk about numbers.
I have mentioned before, that Samsung offers plenty of information on their developers' website.
I am about to present it to you in more details.

There are "Accounting" and "Statistics" tabs in their site's main UI.

I think I have ranted about "Statistics" enough already (see my older posts). But, for completeness, check out how my "daily downloads" look like today:


"Accounting" has 3 sub-tabs - "Sales Summary", "Financial Report" and "Settlement Report".

As I mentioned in my previous post, "Sales Summary" gives you detailed information about your sales, but does not include any totals. According to Samsung:
  • Sales Summary is a monthly performance report that updates daily the revenue through SAMSUNG Galaxy Apps and the income of the seller.
    Although the report can be queried monthly, the data are updated daily.
    Therefore, the users can query the updated performance everyday.
So I made a little program to summarize the data and present it in a decent way. I downloaded the data once per day since June 14th. Today it looks like this:


"Updates daily", my ass. More like "every couple of days, at random times".
Now if someone can explain me the relation between these numbers and the ones in the "Statistics" image before - be my guest. I am pretty sure neither of them reflect the truth.

Here is how my "Financial Reports" look like for April and May:





Now, time for observations:

- "Sales Summary" (remember, this is their data, my app just displays it properly) says $211 for April and $466 for May. "Monthly Reports" (their other data) say $88 for April and $400 for May. Cool, huh? I actually have an explanation for this - it looks like "Financial Reports" do not cover the last week (or so) of the month. Since I released my app on April 19th, the differences for April are big.

- If you sum "Financial Reports" for April and may you get $489.76. The "Settlement Report" says $472.70. Pretty close, but not the same. Again, financial reports don't reflect the last week of the month, and maybe Settlement report does? But if it did it should be $211 + $466, shouldn't it? More likely, it is like the financial reports, but minus some payments which are not yet received by Samsung (they say some mobile operators can pay as late as 6 months after). I don't know.

- Oh, and just to complete the picture, according to their "Statistics", here are the monthly sales totals before taxes and commissions: April: $282.45; May: $628.65; June: $714.73 (Total: $1625.83).

So, Samsung generously provide me with at least 4 different ways to get some information about how my app is doing. And they all show 4 different results! At the end, all I get is confusion and headache.
If they only gave me one, I would have no other option but to accept it as "the truth" and move on (and several of my blog posts would not exist).

The real truth is: Their data (or at least the way it is presented) is a big fucking mess, just like their software.

Monday, June 22, 2015

Internet connected LCD

A while ago I told you about my smart home setup. Part of it was a cheap Nokia 3310 LCD displaying the time and room temperature. Then I told you about the ESP8266, a microcontroller with built in WiFi.
Today, after a lot of struggling, I finally managed to combine these 2 together. Now I have an internet connected LCD:

Two ESPs died in the process - no idea how, they just died. RIP. Its time to order some more, because I have only one left.

Originally, I used the Adafruit library on the Arduino. So, the first thing I tried was to try to compile it for the ESP. Unfortunately, it didn't compile and at that time porting it seemed non-trivial. So I looked up for alternatives. And found out here that these examples should just work on the ESP.
The first problem is that these examples require 5 GPIO pins and the ESP-01 only has 4.
But, in the description of this library it says you can connect Chip Select to ground (because I don't have other devices sharing the SPI bus) and Reset to the Reset of the ESP. Which would mean it would only take 3 GPIO pins and leave me 1 spare!
So, I tried, compiled the examples and ... they didn't work. Just a blank screen. I double and triple checked my connections, my code ... everything was fine.
In desperation I tried the same examples, but with the Arduino instead with the ESP. Surprise, surprise, they didn't work on the Arduino either.
So I took the Adafruit's library and example (which work on the Arduino) and started comparing it with the code I was using.
Ultimately, I found out the following:
- You have to call "set contrast" again after initialization of the LCD for some reason (the initialization itself sets it). No idea why. But that was what caused the playground samples not to work. Maybe it is only the Chinese LCD + breakout I am using.
- It is indeed OK to connect "chip select to ground"
- It is NOT OK to connect Reset to the reset pin of the ESP. It needs a reset after the ESP has started. I am guessing it is because the ESP outputs stuff on its pins on boot, which might confuse the LCD.

So, for now I have 2 options - to waste the 4th GPIO pin with the sole purpose to reset the LCD at startup and stay HIGH from then on ... or to somehow reset the LCD by other means.
For the moment I just added another push button to reset the LED manually and made the code wait for 2 seconds on startup (so I have time to press it). It is far from ideal - as I have said before, finished projects should only have an on-off switch and that's it.
I believe it is possible to replace the push button with or more 555 timer chips, I just have no idea how ... yet.

Still, I now have an internet connected LCD and a spare GPIO pin. I have no idea how I am going to use them, but it is cool by itself.

The next step would be to actually port the Adafruit library - I now know enough about its inner workings, so it shouldn't be a problem.


Saturday, June 20, 2015

Back to the ... bread ... board!

In my previous post I was excited to let you know about the ESP8266 breakout I did. And I'm still proud of it.
But then I discovered two things:
- You can use the TX and RX pins as GPIO pins! This means you can youse 4 GPIO pins on the cheapest ESP8266-1 module, not 2. That makes it even more useful (you can control up to 4 things over the internet now, not just 2, using a single ESP board).
- The ESP is incredibly picky. It doesn't like when stuff is connected to its GPIO pins when its booting. If you have 4 LEDs connected to its GPIO pins the chance is it won't boot. And, it depends on the type of the LEDs. Red ones tend to cause most trouble. I am not joking.

So, my cute little breakout is now a bit obsolete - it is still useful, but not ideal.
I was presented a new challenge - to find a convenient and reliable way to use all 4 pins on the smallest ESP.
The problems:
- When booting, all 4 GPIO pins should be floating
- When in "programming mode" GPIO0 should be low, RX and TX should be connected to USB2TTL
- When in "running mode", all 4 pins should be connected to whatever I want to connect them to (LEDs, motors, relays, sensors, etc.)
- As soon as uploading a new program is finished, the ESP goes to "running mode" and starts executing that program. This is while its pins are still in "programming mode" configuration.
- I need a "development setup" setup where I can repeatedly toggle between boot/programming/running mode (so I can upload and test new code) and a "final setup" for the completed projects (it still has to boot with all pins floating and later toggle them to "running mode").

Long story short, my "development setup" looks like this:
After buying the proper switches (called "2P3T switches) it was fairly easy. Each switch controls 2 independent circuits and has 3 different positions. The only problem was that these switches were not breadboard/protoboard friendly, so I had to make little "breakouts" for them. Special thanks to a friend of mine who cut the PCBs for me, to make it possible.
Now I have the usual reset button, power on/off switch and two 3-state switches. These 2 toggle 2 pins each between programming/boot/running mode. The left one is for GPIO0 and GPIO2 and the right one is for TX/RX (a.k.a. GPIO1 and GPIO3). I used 2 separate switches so I can control TX/RX separately. So, I can still use them as TX/RX (e.g. for debug output to PC during development).

The problem that the program starts running as soon as the upload is completed turned out to be not a real problem. At first I was afraid outputting HIGH through RX/TX pins could cause some damage (to the USB2TTL board or to my laptop's USB). There was one way to find out - I put on protective glasses, helmet, had a fire extinguisher ready, crossed my fingers and just tried it. Nothing bad happened. You can even see the USB2TTL's lights blinking when my program sends high/low through TX/RX. I only had to use a pull down resistor between GPIO0 and ground instead of simple wire, because outputting high signal to GPIO0 (which is connected to ground in programming mode) resulted in a short circuit. Fortunately, it didn't blow the ESP, just made it reset.

So, now developing for the ESP is easy - i put the 2 switches to "neutral", turn the power on, then switch to "programming mode" (both switches to the left), upload a new program, flip one or both switches to the right when the program is running if needed. For corrections, I just flip the 2 switches back to the left, hit the reset button and upload the new program. It's not as easy as with the Arduino, but it's easy enough.

Now, the "final setup" is a bit more tricky. I could probably go with a few 2-state switches - an on/off switch and 1-2 switches to toggle the GPIO pins between "boot mode" and "running mode". But that would be kind of lame. Final products only need to have an on/off switch and that's it. And, when you turn it on, it would need to boot the ESP with all GPIO pins floating and then reorganize its pins to "running mode". After some fiddling and with the help of a 555 timer chip and 2x relay chips I came up with this prototype:

Well, right now there is no "on/off" switch, you just plug the battery, but that's just because the switches I have do not fit the breadboard very well and cause bad connections. When soldered on a proto board they will be a not-an-issue.
What happens now is when I turn the power on the ESP boots and after a second or so the timer kicks in, turns the relays on and they switch the 4 pins into "running mode". Smooooth :)

Btw, these 555 timer chips are awesome little toys, they can do so many things. And they cost like $0.75. For example, I plan to use one to make a horn for my electric scooter and another one to make blinkers for it. If you're into electronics and don't know what a 555 chip is (as I didn't until few days ago) - look it up, there is plenty of info and plenty of different usages.

Oh, and the current test makes the ESP create a WIFI hotspot, you connect to it (from a phone/laptop/tablet), and you can control the 4 LEDs from a web browser. I also tried connecting the ESP to my router instead of creating a hotspot - no problems.

So, now I can control 4 pins on a single ESP82266-1 from anywhere in the world :)
This means I could turn 4 lights/appliances on/off, drive an RC car (two motors in both directions) and whatnot ... over the internet. Using components that cost like $5 total. Nice, huh?

The next step is to turn the above prototype into a much more compact breakout, but I am not in the mood for tricky soldering today ... and I'd like to wait a little longer before I do that, just in case I get some other ideas in the meantime.




Monday, June 15, 2015

Screw Samsung, I am doing it myself!

In my previous pos I told you how bad Samsung's Seller Office is (the site that developers are supposed to use to add/update apps, view downloads/sales statistics, buyer comments, etc.)

One thing I complained about, was that there was no easy way to tell actually how much your app has sold. There are "Statistics" and "Accounting" sections on this site.

Statistics break way too often - in my last post I complained they are broken for the 3rd time, guess what - they are still broken.
When they work, they don't reflect all the data (I have proven that in my last post) and the numbers they give you are "net" sales - before Samsung cuts their 30%, some countries take VAT, some mobile carriers take a percentage too. At it varies from country to country and from carrier to carrier.
Still, when they work, they are somehow useful as a rough estimation.

Accounting data seems to be more accurate. I guess they had the common sense to put someone with at least half a brain in charge of accounting data. But whoever designed the features on the site ... well, let's say he is not the brightest star in the sky. Not by a long shot.

So, until now, I had to choose between Statistics that don't work and accounting data that looks accurate but is too raw - too much useless information and no overview (e.g. no "Total" fields).

There is one way though - in the accounting data, there is this button labeled "Excel". When you click it - surprise, surprise - it downloads an excel file, containing the same raw information. It takes some search-and-replacing to transform the data they give you to something Excel actually recognizes as numbers. But then, you can select the last column and, lo and behold, you get a total number of dollars. For that month. Repeat this for all months, sum the totals and you get the total amount of money your app has made! It takes less than half an hour to get this number! I really wanna slap someone.

But yesterday, after statistics have been broken for 4 days, I thought - "Hey, wait a minute! The information is there, in the accounting excels. Yes, it takes some fiddling to get to the totals, but this process can be automated. Like, what computer programs do. And I happen to create computer programs for a living, as well as for fun."

So, I created a computer program :)
It looks like this:



It scans my downloads folder" for xls files, parses them, recognizes the ones containing Samsung's sales information, extracts the data and presents it to me in the shape I want it.
That is - with grand total, monthly totals and "daily" totals. Well, technically it is not "daily", it is "between 2 samples", but if I download this excel every day, it is daily. It also gives me per-country breakdown for each of these categories. It even works fast - I haven't measured it precisely, but it takes less than half a second from double-clicking the shortcut icon to presenting the data. Pretty neat.
Oh, and it took me a couple of hours to do it, and that is because I have very little experience with C# (I chose to do it with C# for learning), so I had to google how things are done like every 5 minutes.

I wish I could scrape the info from their website, instead of having to manually download the excel files, but I can't - there is a login I don't know how to get past from my program (i.e. how to simulate user/pass entry). But it is not really a big deal - once a day (with my morning coffee) I have to log in, download the excel and run my program. And I check the site often anyway, to check for buyer comments, because they don't have the option to send me an e-mail when a new one arrives (or an old one is edited), so I have to poll.
Ideally, if I figure out how to get past the log in, I could make my program get the sales data AND buyer comments, analyze it, notify me of any changes ... and have it displayed on my smart home server, just because I can :)

Now, I can only hope Samsung's accounting data is indeed accurate and updated daily.

For the moment, it looks like it is updated way more often than once a day - this morning I got different data in several tries within less than 2 hours.

What is a bit alarming is the numbers it shows - there is no way I have done so many sales for one day!  At least according to their statistics, my app usually sells between 10 and 25 copies a day, with rare "peaks" to 30+. For my app to make $100+ it has to have around 100 downloads (it currently costs $1.99 and almost 50% of them go for taxes and commissions).
So, there are 3 options:
1: My app suddenly started selling exceptionally well, just after I made the program to track it - doesn't sound very likely, does it? This only works in the field of quantum physics :)
2: The data actually reflects more than 1 day (since today is Monday, it could be reflecting the whole weekend) - this sounds most likely to me
3: There is a bug in my code - completely possible, but looking at the raw data confirms the results.

Anyway, time will tell.

By the way, as I am looking at the data they give me more carefully now, I am starting to realize what a big mess it is. I think it deserves a separate post with many examples and pictures, so stay tuned.