Connecting SKU193431 I2C RTC To a Raspberry Pi

The I2C module came in a bag with "SKU193431 EVU" on the sticker, which Googling finds the banggood page from which it was ordered, but not much else.

http://www.banggood.com/RTC-Real-Time-Clock-Module-For-Raspberry-Pi-p-965976.html

There is no schematic or any further details, also there is no markings on the RTC module to indicate even which pin is positive. I found the following image on the internet showing the GPIO header pinout:



I took a guess that I should connect it to point 01, 03, 05, 07, 09. This is the inner most set of connectors. Then I carried on with a guide that I had found online. Luckily my guess was correct. There are no other pins on the GPIO header with an I2C label, but I was mostly concerned about the power connections - if connected the wrong way with 3.3V the wrong way across the battery it could cause some damage. Anyway, luck was on my side and it seemed to all work fine. So on to the software setup.

The first thing to do is to enable I2C in the raspi-config tool. To do this, open the configuration tool:

sudo raspi-config

Go in "Advanced Options" > "I2C" > "Yes"

When asked if the default module should be loaded by default, select "Yes", and then reboot the raspberry pi.

Then I followed steps from here: http://www.drewkeller.com/blog/adding-hardware-clock-raspberry-pi-ds3231

# Comment out the blacklist entry so the module can be loaded on boot
sudo sed -i 's/blacklist i2c-bcm2708/#blacklist i2c-bcm2708/' /etc/modprobe.d/raspi-blacklist.conf
# Load the module now
sudo modprobe i2c-bcm2708
# Notify Linux of the Dallas RTC device (use -0 for Model A or -1 for Model B)
echo ds3231 0x68 | sudo tee /sys/class/i2c-adapter/i2c-1/new_device
# Test whether Linux can see our RTC module.
sudo hwclock

The response was the time that the RTC thought it was, which hadn't yet been set. If you get this, then that looks good ad the hardware aspect is correctly connected and working.

The raspberry pi was connected to the internet and got the time and date automatically, as can be checked by the "date" command.

If the date is wrong, this can be set with the following command:

sudo date -s "Jan 27 2016 12:41:01"

I have previously had some problems when setting the time and date together, and found I had better luck setting first the date, and then the time, rather than both together in the same command. I think this was due to problems with the formatting, but this method of splitting the time and date helped. Then write the current date and time to the RTC module using the following command:

sudo hwclock -w

Then edit the /etc/rc.local file with this command:

sudo nano /etc/rc.local

and add the following two lines before the line "exit 0":

echo ds3231 0x68 > /sys/class/i2c-adapter/i2c-1/new_device
hwclock -s

Now disable the fake hardware clock with the following line:

sudo update-rc.d fake-hwclock disable

The network time can also be disabled, but I wanted to leave this in place. To disable this, use the following command:

sudo update-rc.d ntp disable

But if it is left in place, then the clock should be updated each time that the raspberry pi connects to the internet, but will keep time when powered off, and when rebooted without power.

Python base64 decode Problem

I was struggling with a problem in Python, trying to use the base64 package to decode some base64 data, using the base64.decode() function. I was passing my string to the function, so my code looked link the following:

import base64
#code to load base64 data from file
encoded='ZGF0YSB0byBiZSBlbmNvZGVk'
outputdata=base64.decode(encoded)

Well I obviously had different code to handle the loading of the data from the file, but I have just changed the code to use a string with encoded data for this example.

If you try and run this code, you will get the following error:

Traceback (most recent call last):
  File "removed", line 4, in <module>
    outputdata=base64.decode(encoded)
TypeError: decode() takes exactly 2 arguments (1 given)

This was confusing as I couldn't find any more information about what should be in the second argument, so I tried a few different things, nothing works.  If you do this, the error will instead change to the following:

Traceback (most recent call last):
  File "removed", line 4, in <module>
    outputdata=base64.decode(encoded, '')
  File "C:\Python27\lib\base64.py", line 303, in decode
    line = input.readline()
AttributeError: 'str' object has no attribute 'readline'

The problem was that I was infact calling the wrong function name. There are 2 separate functions, and the function I thought I was using, takes just a string, and the function that I was actually using takes 2 parameters, and I think is intended to read from a file on this disk.

So this is quite an easy fix, which is to instead use the function base64.b64decode(encoded_data), and just means I wasn't reading the manual well enough. I am documenting it here because Googling the previous error messages along with some keywords about base64 and decoding didn't turn up anything. It was only when I went back to the very basic example that I noticed my error. Thus the final working code is as follows:

import base64
#code to load base64 data from file
encoded='ZGF0YSB0byBiZSBlbmNvZGVk'
outputdata=base64.b64decode(encoded)
print(outputdata)

Creating a captive portal on the raspberry pi that functions separately to the ethernet connection using a Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter

I wanted to add a USB WiFi dongle to my Raspberry Pi to allow connection to it with a mobile phone, and to display a status page in the browser, but I didn't want to bridge the WiFi network to the internet, so that I didn't have to worry about unauthorised people accessing the internet through my WiFi connection. I also didn't want to disconnect the Raspberry Pi from the internet completely, as it it "phones home" to my server occasionally so that I know it is still running, and by leaving the ethernet connection functioning fully, it also allows me to update the Rasbperry Pi or make changes to the software.

Most of the tutorials on the internet for setting up a wireless access point (AP) are assuming that it will done with the connection to the internet. This is not the case for me, and so by some trial and error, and following a few different guides, I have finally found a working solution.

The first step was to understand the hardware that I had. Using the command "lsusb" lists the USB devices that are connected. By entering the command, then connecting the USB WiFi dongle, and entering the command again it is easy to see what has changed. For my dongle, I get the following information:

Bus 001 Device 005: ID 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter

The identifying string is RTL8188CUS and googling this shows some issues with the driver when trying to connect it to the raspberry pi.

First of all, I got the dongle working as an access point with the correct driver. To do this, I followed the excellent tutorial by Dave Conroy. The tutorial can be found here: http://www.daveconroy.com/turn-your-raspberry-pi-into-a-wifi-hotspot-with-edimax-nano-usb-ew-7811un-rtl8188cus-chipset/

I already had the raspberry pi running correctly, so I started with the "Prerequsites" section. The first command to install two packages is as follows:

sudo apt-get install bridge-utils hostapd

With this command, both bridge-utils and hostapd were installed. I don't think bridge-utils was needed, but I left the command incase there are any changes made by this package that weren't immediately apparent.

The next set of commands were copied verbatim from Dave's Blog. He hosts a replacement hostapd binary which resolves the incompatibilities with the WiFi dongle chipset. This can be downloaded and installed with the following commands:

wget http://www.daveconroy.com/wp3/wp-content/uploads/2013/07/hostapd.zip
unzip hostapd.zip
sudo mv /usr/sbin/hostapd /usr/sbin/hostapd.bak
sudo mv hostapd /usr/sbin/hostapd.edimax
sudo ln -sf /usr/sbin/hostapd.edimax /usr/sbin/hostapd
sudo chown root.root /usr/sbin/hostapd
sudo chmod 755 /usr/sbin/hostapd

The above commands will replace the standard hostapd binaries with new updated ones from the zip file that are compatible with out WiFi dongle driver.

The section on "Bridging the Connections", I skipped entirely, as I want the ethernet and Wifi networks to be separate. I skipped to the section "Configuring Hostapd". Here to enter the file editor, enter the command:

sudo nano /etc/hostapd/hostapd.conf

Then there is some recommended entries for the file, however I modified this somewhat. My version of the file is as follows:

interface=wlan0
driver=rtl871xdrv
#bridge=br0
ssid=NETWORKNAME
channel=6
wmm_enabled=0
#wpa=1
#wpa_passphrase=ConroyPi
#wpa_key_mgmt=WPA-PSK
#wpa_pairwise=TKIP
#rsn_pairwise=CCMP
#auth_algs=1
#macaddr_acl=0
ctrl_interface=/var/run/hostapd
ctrl_interface_group=0

The hash (#) character represents a comment for the entire line, and so the line is skipped over. I have commented out the section to do with security as I want the hotspot to work without authentication. This is also why I wanted to separate the two networks and not bridge them. The last two lines were added in from a separate tutorial, which I will link to shortly.

The next step was to reboot, and then test the configruation using the following command:

sudo hostapd -dd /etc/hostapd/hostapd.conf

Then I checked with a WiFi enabled device and found my network SSID. This allowed me to connect, however requesting a website failed - which for me was a success!

Now I started to Google further for how to make a captive portal, and found a guide on the website sirlagz.net, here: http://sirlagz.net/2013/08/23/how-to-captive-portal-on-the-raspberry-pi/ - This guide also has several previous steps on setting up the access point. So I went back to the access point guide and reviewed this to make sure I hadn't missed any prerequisits here also. This guide is linked here: http://sirlagz.net/2012/08/09/how-to-use-the-raspberry-pi-as-a-wireless-access-pointrouter-part-1/

Something that I hadn't done yet was to assign a static IP. To do this, enter the following command;

sudo nano /etc/network/interfaces

This opens the text editor for the network interfaces. Then further down the file, there should be an entry for wlan0. Change the entry so that the settings are as follows:

iface wlan0 inet static
address 10.0.0.1
netmask 255.255.255.0

My complete interfaces file is as follows:

# interfaces(5) file used by ifup(8) and ifdown(8)
# Please note that this file is written to be used with dhcpcd# For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'
# Include files from /etc/network/interfaces.d:source-directory /etc/network/interfaces.d
auto loiface lo inet loopback
iface eth0 inet manual
allow-hotplug wlan0iface wlan0 inet staticaddress 10.0.0.1netmask 255.255.255.0#    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
allow-hotplug wlan1iface wlan1 inet manual    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Note once again, a hash at the beginning of the line indicates the line is commented. Next the hostapd file needs to be set. To do this, open the file with the following command:

/etc/default/hostapd

And find the line with the following text:

DAEMON_CONF=""

This needs to be modified so that it refers to the location of the .conf file, so it should appear as follows:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Then the access point tutorial talks more  about changing the hostapd.conf file, but we did this earlier. Also there is some changes to the WiFi settings, but these are also okay. Finally we need to install dnsmasq. The tutorial I followed mentions that this is to handle the issuing of IP addresses.

To install the package, run the following command;

apt-get install dnsmasq

Then we need to edit the .conf configutation file. To open that use the command;

sudo nano /etc/dnsmasq.conf

This is a huge file, and we need to scroll right to the bottom. The keyboard shortcut ctrl+v scrolls down one page at a time, which is faster than the arrow keys going one line at a time. At the bottom add the following lines:

interface=wlan0
dhcp-range=10.0.0.2,10.0.0.250,255.255.255.0,12h
address=/#/10.0.0.1

For my file, it looked like everything else was commented out. It is a long file and I didn't look so closely, but if everything else is commented out this shouldn't actually need to be at the bottom of the file. Anyway, the line configures the wlan0 interface and sets the range of IPs that can be given out. In the original tutorial, this is a much smaller range, I expanded this to cover more IPs. The last line is taken from the final tutorial that I followed, which redirects all requests to 10.0.0.1. This is the IP address of the raspberry pi on the wlan0 interface, as set earlier on in the /etc/network/interfaces file, and must match this IP. So now after saving the file, any traffic on the wlan0 network should be redirected to the raspberry pi.

The final tutorial that I followed is also from sirlagz.net, and is linked here: http://sirlagz.net/2013/08/23/how-to-captive-portal-on-the-raspberry-pi/ and talks about setting up the captive portal. The only real change to be made was changing the last line of the dnsmasq.conf file, as done a few lines above.

Now the only remaining thing to do is to install a webserver. The server suggested in the sirlagz tutorial was lighttpd, which I have also used at the moment. To install this, use the following command:

apt-get install lighttpd

Finally I restarted everything for good measure. Now connecting to the wifi network and requesting a page, redirects to the Placeholder Page for lighttpd. Success! The location for my files that are served up by the webserver is:

/var/www/html/

Within this folder, is the location that the files need to go for the webserver.

Update: I had rebooted several times along the way, and thought that the above worked fine, but trying it the next day had problems, and no longer worked correctly. There are a couple of half related issues. Pinging google.com would resolev to my local ip of 10.0.0.1, but pinging 8.8.8.8 for example would fail completely. The problem with pinging ip addresses was related to the gateway not being setup correctly. To see this, enter the command "route -n". Here, to work correctly, there needs to be an entry in the "Desitnation" column with 0.0.0.0, and with your gateway in the "Gateway" column. This wasn't working for me, and so I can use the following command to add the gateway correctly:

sudo route add default gw 192.168.100.1 eth0

Where  my gateway IP is 192.168.100.1, and the interface is eth0. I haven't yet found a way to get this to work automatically. It would be possible to add this to a script without any problems, however I don't know how the gateway could be changed to the correct IP for the network.

The second problem was with DNS lookups. To fix this, I needed to add a couple of lines to the /etc/dnsmasq.conf file. I had thought that the dnsmasq captive portal only applied to the wlan0 interface, but this doesn't appear to be the case, and it overrides traffic from all interfaces. So I made two changes. The line:

address=/#/10.0.0.1

I changed to the following:

address=/pi/10.0.0.1

This now only redirects the domain "pi" to the local server, not all addresses. I also added the following two lines to the end of the file:

server=8.8.8.8
server=8.8.4.4

This uses the Google DNS servers for DNS lookups. After running both of these commands, it should be now possible to ping google.com, and to perform updates etc. So it appears that all the settings apply to both interfaces.

Update 2: After testing on another network, it seems that the problem with the default gateway being added only occurs on one network, that is perhaps configured incorrectly, and it isn't actually a problem with the setup on the raspberry pi. I found this out by reinstalling raspbian again and not following the above steps, and leaving all of the network connection settings as the default. Then it was still not possible to connect to the internet automatically. This is strange as it wasn't previously a problem, and I never used to have the problem where it was necessary to add the gateway. However it seems something in the network has changed, which was unrelated to what I had done, but I had falsely attributed it to the steps I had taken. So the "update" comments can be ignored.

Code Snippet: Ping local IP addresses

Here is a code snippet that I have created for the purpose of finding a Raspberry Pi when it is connected to a network without a screen. For my setup, I have both an ethernet connection and a WiFi connection. Normally the ethernet connection is connected directly to the raspberry pi, and the WiFi connection goes directly to the router. My code was written and tested on Windows 8.1, with Python 2.7.10. 

from __future__ import print_function

import socket

import os

import time

import subprocess

from threading import Thread

def ping(hostname, timeout=100):

    timeout=str(timeout)

    #perform ping command. -n sets the number (1 here) and -w sets the timeout.

    #creationflag=8 hides the window.

    response = subprocess.Popen(["ping", hostname, "-n", '1', "-w", timeout],

                                stdout=subprocess.PIPE, creationflags=8).stdout.read()

    ping_reply=0

    if "time=" in response.lower() or "time<" in response.lower():

        ping_reply=1

    return ping_reply

def ping_and_print(hostname):

    #perform ping command

    ping_reply=ping(hostname)

    #print whether ip address responded to ping or not.

    #I added the "\n" character to the print command because

    #when multiple threads try to print, it was causing

    #overlapping for some reason. It seems there is a bit of a delay

    #between the string being printed and the end of line

    #character being printed.

    if (ping_reply==1):

        print(hostname + " - UP!\n", end="")

    else:

        if verbose==1:

            print(hostname + " - Down!\n", end="")

    return

verbose=0 #also print out ips that aren't responding to pings.

#this gets the local ip range, the range with the ethernet adapter

local_ip=socket.gethostbyname(socket.gethostname())

#this pings the gmail server and gets the IP address used to connect, so

#should always be the ip address used for internet connections

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

s.connect(("gmail.com",80))

web_facing_ip=s.getsockname()[0]

s.close()

print("Local IP     : " + str(local_ip))

print("Web facing IP: " + str(web_facing_ip))

#check we can actually ping the internet

hostname="8.8.8.8"

ping_reply=ping(hostname,5000)

print("Internet ping: " + str(ping_reply))

#create the network stem for the local ip range

network_stem=".".join(local_ip.split(".")[0:3])

#ping primary ip range

for ip in range(1,255):

    this_ip_address=network_stem + "." + str(ip)

    t = Thread(target=ping_and_print, args=(this_ip_address,))

    t.start()

#create the network stem for the internet connecting ip range

network_stem=".".join(web_facing_ip.split(".")[0:3])

#check we are running on a second network range and not just repeating the

#previous tests.

if local_ip<>web_facing_ip:

    #ping secondary ip range

    for ip in range(1,255):

        this_ip_address=network_stem + "." + str(ip)

        t = Thread(target=ping_and_print, args=(this_ip_address,))

        t.start()

#sleep at the end for a bit to wait for remaining responses.

for i in range(1,100):

    time.sleep(0.01)

print("Done.")

Matlab minimum for variables of different dimensions with the min() function

When using the Matlab function min() to find the minimum value of a variable, I wanted to use it for data that was in more than just 1 dimension. I received the following error message when running my code:

Error using min
Matrix dimensions must agree.

The problem here is that I was trying to find the minimum of several variables together, and the variables each had different dimensions.

Furthermore, the function acts somewhat differently to what I would expect. Consider the following initialisation:

>> data=[[1:10];[1:10]]
data =
     1     2     3     4     5     6     7     8     9    10
     1     2     3     4     5     6     7     8     9    10

Here the data fills a 2 x 10 matrix. I would expect the min() function to return one value for the entire matrix. Instead, the min() function returns the minimum for each column.

>>  min(data)
ans =
     1     2     3     4     5     6     7     8     9    10

One solution is to use multiple min() calls as follows:

>> min(min(data))
ans =
     1

But this isn't a great solution, as you might require this multiple times. So the solution is to use (:) to reference the elements of the matrix. This references all of the elements consecutively, and results in a 1 dimensional matrix containing all elements.

data(:)
ans =
     1
     1
     2
     2
     3
     3
     4
     4
     5
     5
     6
     6
     7
     7
     8
     8
     9
     9
    10
    10

As the data is now one dimensional, the min() function will work on the data in this format, as follows:

>> min(data(:))
ans =
     1

This is how I would have expected the min() function to behave by default, but at least this works for one variable. However if I now want to use it to find the minimum from multiple variables, again the behaviour starts to cause a problem. I will create two 2x10 matrices for testing as follows:

>>  a=[[1:10];[1:10]]
a =
     1     2     3     4     5     6     7     8     9    10
     1     2     3     4     5     6     7     8     9    10
>>  b=[[11:20];[11:20]]
b =
    11    12    13    14    15    16    17    18    19    20
    11    12    13    14    15    16    17    18    19    20

If trying to find the minimum by passing both variables to the min() function, this again doesn't work entirely as expected.

>>  min(a(:), b(:))
ans =
     1
     1
     2
     2
     3
     3
     4
     4
     5
     5
     6
     6
     7
     7
     8
     8
     9
     9
    10
    10

The reason for this is that Matlab sees the two variables as two columns of data that have been joined together, and then passed to the function. Matlab processes the data as follows:

>> [a(:),b(:)]
ans =
     1    11
     1    11
     2    12
     2    12
     3    13
     3    13
     4    14
     4    14
     5    15
     5    15
     6    16
     6    16
     7    17
     7    17
     8    18
     8    18
     9    19
     9    19
    10    20
    10    20

So we can see that the min() function actually finds which dimension is longest, and finds the minimum value in the shorter dimension. If we transpose the matrix first using the apostrophe, and then use the min function, the function is still operating on the longest dimension.

>> [a(:),b(:)]'
ans =
  Columns 1 through 18
     1     1     2     2     3     3     4     4     5     5     6     6     7     7     8     8     9     9
    11    11    12    12    13    13    14    14    15    15    16    16    17    17    18    18    19    19
  Columns 19 through 20
    10    10
    20    20
>> min([a(:),b(:)]')
ans =
  Columns 1 through 18
     1     1     2     2     3     3     4     4     5     5     6     6     7     7     8     8     9     9
  Columns 19 through 20
    10    10

There are a couple of obvious workarounds for this problem. The first would be to use multiple min() function calls, as follows:

>> min(min(a(:),min(b(:))))
ans =
     1

This isn't a particularly nice line of code, with three min() function calls. An alternative that I find better  can be created by appending the data before calling the min() function, and is as follows:

>> min([a(:); b(:)])
ans =
     1

Care must be taken to use the semicolon in between the matricies to minimum value, if using a column this will create two columns of data, and the min() function then returns two values as follows:

>>  min([a(:), b(:)])
ans =
     1    11

I'm not sure why in this case the min() function is now only returning two values, and not working in the longest dimension like it was previously.

Alternatively if using the comma to separate variables, the variables can be trtansposed first, in which case the function works, but this is more work for the same result and so isn't recommended.

>> min([a(:)', b(:)'])
ans =
     1

Summary

The behaviour of the min() function is to find the minimum value in a given matrix, not the absolute minimum in the data passed to the function. To get the function to behave in this manner requires some changing of the data format before the data is passed to the function. I believe the cleanest format is the following:

>> min(min(a(:),min(b(:))))
ans =
     1