Basic Network Programming (Python on Symbian)
From Symbian Developer Community
This chapter provides an overview of key IP networking concepts and the Python sockets API.
The Advanced Network Programming extends the concepts in this chapter to discuss higher level networking protocols (e.g. HTTP, XML-RPC) and to cover the essentials of multi threaded network programming.
Contents |
Introduction
Adding networking capability to an application can vastly increase its interest and value, allowing users to play, communicate, and share information with their friends and business contacts and to access the vast resources of information available on the Internet.
Python is a great runtime for writing networked applications, making it possible to construct network servers and clients with just few lines of code. Python on the Symbian platform shares this benefit - there are a few different paradigms associated with mobile devices (and consequently some different socket options), but in general porting from the desktop is a straightforward exercise.
The chapter is accompanied by worked examples showing how to set up simple ad-hoc networks with a client and server, and how to use multicast and broadcast networking. In addition, the chapter briefly covers how to use exception handling and other debugging techniques in your networked application.
Basic Network Principles
IP Addressing
Network programming is innocuous if you don\'t have at least some basic knowledge about IP addressing. In IPv4 (Internet Protocol version 4) any networked device is addressed by a 32 bits number. This number is commonly represented as a sequence of four decimal numbers, each one representing one byte. These four decimal numbers are separated by dots for better understanding. For instance, valid IP addresses could be 200.201.40.5 or 10.20.30.40. Since each number is representing one byte (8 bits), they may vary from 0 to 255. As we will see, some combinations are not valid for addressing machines and there are some reserved IPs.
We have three types of addresses in IPv4: unicast, multicast and broadcast. Unicast is the typical situation that we usually call point to point connection. In this case you want to connect only to a specific target and this target must have an exclusive address. Several protocols uses unicast addressing, like when you are surfing on the Internet (HTTP), reading your email (SMTP) or transferring a file via FTP. However, if you want to write only one message and address it to several machines, you might use a multicast or broadcast address. Multicast addresses are tied to the concept of a group of machines, not to a specific machine address. When you use a multicast address, you can reach all machines in that group. Machines that want to belong to a group must do it explicitly, joining to the group via a socket API call. The group is, in fact, an IP address from a reserved range, as I will see later. Finaly, Broadcast addresses are similar to multicast but less selective: the messages are sent to all machines (no subscription required) in a specific subnet or in the current local area network, if the subnet is not specified.
It is easy to understand how the IPv4 space addressing was divided when we use the IP Addressing Circle. If you consider that all possible IPv4 addresses are in the circle, half of these addresses starts with 0 in the most significant bit of first byte (0XXXXXXXb). This define what we call class A IP addresses. In the other half, we have only bytes with 10 or 11 in the first two most significant bits. The quarter starting with 10XXXXXXb is called class B IP addresses and the other quarter is divided again using the same logic among class C IP address (110XXXXXb), class D IP address (1110XXXXb) and class E IP address (1111XXXXb). Classes A, B e C are used for unicast, class D is used for multicast and class E is reserved.
With this division rule in your mind, we can calculate all IP address ranges just inspecting the first byte. For instance, for class A (0XXXXXXXb), the first number is when we put all X's in zero and the last number when we set all X's to one. In this case, we will have 00000000b and 01111111b or 0 and 127, in decimal. The following table summarizes these ranges.
Got it ? When you see an IP address like 150.164.34.99, this IP belongs to class B network. And it is simple to detect just observing the first number. Each unicast address (Class A, B and C) is divided in two parts, known as network ID and Host ID. Depending on the class, a different number of bits is allocated to each one. Three differents partition were created to allow a variable number of hosts in a network, as depicted below.
The reasoning in this division is that machines in the same network must have the same Network IDs and, of course, different Host IDs. For instances, machines 10.10.23.1 and 10.100.55.3 belong to the same network (10) and have different host IDs (10.23.1 and 100.55.3). However, this kind of division was inefficient, generating a poor usage of the space addressing and a new partitioning scheme was created: CIDR (Classless Inter-Domain Routing), explained later.
Subnet masks
Before some more practicing, it is time to finish the last post about IP addressing. One important concept is missing: network masks. After CIDR adoption, the network mask became essential for splitting the IP address in network ID and host ID. This way, the default masks for classes A, B and C could be increased (typical case for sub-networking) or even decreased (common in routers for saving routing tables entries).
In fact, the network mask is built putting bits in 1 from left to right in the part corresponding to the network ID and leaving the remaining bits in 0 (host ID part). For instance, consider the class A network address 10.0.0.0. We will use the network mask 255.255.255.0. This mask has 1 in the first 24 bits (three consecutive bytes with 255) creating the network ID 10.0.0 and leaving one additional byte for host ID. The CIDR notation for this mask is /24, an abbreviated expression to say the amount of bits set. Different machines in this network will have distinct host IDs, just changing the bit values in the host ID.
Two host IDs are reserved: the first one (all bits in host ID reset) and the last one (all bits in host ID set). As you have already guessed this first address is the network address but the last one is new in our discussion. It is called broadcast address and it is reserved for sending messages for all machines in this subnet.
In summary:
Network: 10.0.0.0
Mask: 255.255.255.0 or /24 (CIDR notation)
IP addresses range: 10.0.0.1 to 10.0.0.254 (to be used in your machines)
Broadcast: 10.0.0.255
Let's examine a more complex example. Suppose you received the following network/mask from your ISP (Internet Service Provider):
Network: 200.201.145.128
Mask: 255.255.255.192 or /26 (CIDR notation)
Quite strange no ? In fact, no. I will show you.
Put your attention in the mask. The first two significant bits in the last byte of the mask are set (192 is 11000000 in binary - we will use suffixes 'd' and 'b' to denote bases). This means that the first two significant bits in 128 (10b) belongs to the network ID and only the last six bits can be used to host ID.
So, when all bits in host ID are reset (000000b or 128d) we have the network address and when they are set we obtain the broadcast address (10111111b or 191d).
In summary:
Mask: 255.255.255.192 or /26 (CIDR notation)
Broadcast: 200.201.145.191
IP addresses range: 200.201.145.129 to 200.201.145.190 (to be used in your machines)
How about use this knowledge for creating a adhoc connection between two smartphones ? Really simple. Just select one reserved network, define your mask, choose two different host IDs for IPs and set your security parameters/protocols. The connection setup will require a default gateway. If you do not want to access any network/computers outside your ahdoc network, use your own IP as default gateway. In fact, default gateway is related to routing, defining the path to reach other networks.
Multicast addressing
IP multicast is a special addressing where it is possible to create one-to-many communications. In this case, the same IP package can reach different destinations, saving bandwidth and time. This feature makes an multicast IP address very special since it is not related to any specific network interface but instead to a group of machines. In this case, computers that want to receive data on such multicast IP need to join to this group beforehand. Unfortunately, it is not possible to understand multicast without a previous comprehension about how IP addresses are translated into physical addresses.
Each network interface has its own physical address. This address is called MAC (Medium Access Control). The MAC is a 48 bits number that must be unique for each network interface within a subnetwork. This subnetwork may be understood as a physical network consisting of one or several network segments interconnected by layer one or layer two network devices, like repeaters and hubs (layer one) or bridges and switches (layer two). However, they may repeat if network segments are interconnect by IP routers (layer three). Typical MAC address are represented as six hexadecimal numbers with eight bits and separated by colons, like 00:1E:68:BB:49:AB.
MACs are essential to deliver packages that come from IP layer since network interfaces only deal with physical addresses, not IPs (see figure below). So, in order to translate IP addresses (IP layer) into MAC addresses (data link layer), the ARP protocol is employed. Using ARP requests, tables for translating IP into MACs addresses are created in all networked nodes (called ARP cache), allowing translations and posterior delivery.
But a question arises about multicast addressing: how to deliver a message if we have multiple destinations ? In this case, an special MAC address not tied to any network interface is created from the original multicast IP. Moreover, all nodes that belongs to that multicast group must program their network interface to receive messages addressed to that specific MAC address and not only to their own MAC address. In the next figure is illustrated how the multicast IP 224.202.20.30 is mapped to the MAC address 01:00:5E:4A:14:1E.
Special socket options are used to joint to multicast groups, creating special filters for corresponding MAC addresses in the network interface.
Socket API introduction
A simple program called fortune client/server will be used to introduce the socket API. Instead Chinese fortune cookies, the fortune server will send to its clients a random statement from Zen of Python. I am assuming that you have a mobile and a PC connected to a WiFi router, as depicted below.
Client will run in your mobile and server at your PC. In fact, it could run in mobile as well, but I think it is interesting to introduce clients first and leave servers for further posts. Even with a simple client it is possible to clarify many network programming issues.
Fortune client/server
The fortune client code is below. Take a look on it before reading the comments.
# Fortune client
from socket import *
HOST = "10.0.0.100"
PORT = 54321
s = socket(AF_INET,SOCK_STREAM)
s.connect((HOST,PORT))
fortune = u""
while True:
data = s.recv(1024)
if not data:
break
fortune += data
s.close()
from appuifw import note
note(fortune,"info")
First action is to import the module socket where all functions are encapsulated. Next step is to define where we want to connect, I mean, where the service we want to use is running. When you create a network server, you must define exactly where people can find it. In TCP/IP, this is specified as a combination of IP (Internet Protocol) address and TCP/UDP port. While IP is used to locate a machine in the network, the port is used to locate a service inside this machine. You may run several services in the same machine, just using a different port for each one. The common analogy is to use a company phone number for representing the IP address and an extension number for representing the service, like a department or person.
In fact, TCP/IP uses a third parameter as well, known as the transport protocol. The transport protocol defines how the communication between the two end systems will be performed. TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are the most common transport methods used nowadays. TCP provides a reliable and ordered byte stream between two end systems (similar to make a telephone call to someone) and UDP uses a simpler model based on messages, not byte stream. UDP does not have any order or delivery guarantee (similar to send several letters to someone).
The function socket creates a network file descriptor, very similar to that one received when a file is opened, but only used for network communications. It is important to set which address family you are using and the transport protocol. Common address families are AF_INET (IPv4 addresses) and AF_INET6 (IPv6 addresses). TCP can be selected with SOCK_STREAM and UDP with SOCK_DGRAM. All these constants are located inside socket module.
After socket creation, we are ready to connect to our server. You need to provide a tuple with IP (or host domain name, like symbian.org) and port. Since the transport protocol was set when you created the socket, all elements for localizing the server are prepared. In our mobile phone the connection dialog will be presented and connection will take place. In this example, our server is running on a PC with IP address equals to 10.0.0.100. Probably your local network has different settings so check you PC IP address typing ipconfig (Windows) or ifconfig (Linux) inside some command prompt window. Use this IP in the server and clients scripts.
The fortune protocol is very simple: just connect and wait for your fortune message, no needs for requesting. Other socket function is called for handling the reception, named recv. recv need at least one argument: the maximum amount of bytes to receive at each recv call. When the connection is closed, recv will return an empty string. People erroneously believe that each recv call will receive a complete message or the amount of bytes specified in the call. Take care, TCP is byte oriented, not message oriented. So, network and operating system conditions (delay, buffers, throughput and so on) may change the amount of bytes received at each recv call. Even the streaming is guaranteed by TCP, the programmer must check for his protocol integrity. In our case, we are just receiving bytes and adding them to the fortune buffer while the connection is not closed by server. In this case, the connection is closed by client as well. This line could be omitted, since sockets are automatically closed when they are garbage-collected
The server code is below:
# Fortune server
from socket import *
from random import choice
# Fortune database from The Zen of Python, by Tim Peters (import this)
PHRASES =[ u"Beautiful is better than ugly.",
u"Explicit is better than implicit.",
u"Simple is better than complex.",
u"Complex is better than complicated.",
u"Flat is better than nested.",
u"Sparse is better than dense.",
u"Readability counts.",
u"Special cases aren't special enough to break the rules.",
u"Although practicality beats purity.",
u"Errors should never pass silently.",
u"Unless explicitly silenced.",
u"In the face of ambiguity, refuse the temptation to guess.",
u"There should be one-- and preferably only one --obvious way to do it.",
u"Although that way may not be obvious at first unless you're Dutch.",
u"Now is better than never.",
u"Although never is often better than *right* now.",
u"If the implementation is hard to explain, it's a bad idea.",
u"If the implementation is easy to explain, it may be a good idea.",
u"Namespaces are one honking great idea -- let's do more of those!" ]
HOST = "10.0.0.100"
PORT = 54321
s = socket(AF_INET,SOCK_STREAM)
s.bind((HOST,PORT))
s.listen(5)
while True:
(cs,addr) = s.accept()
fortune = choice(PHRASES)
cs.sendall(fortune)
cs.close()
Run the server script in our PC and execute the client script to test. Some screenshots are below:
As you can see, there is a lot of knowledge even in this very short program. On purpose, several details like exception handling and network configuration were omitted, just to keep the focus.
Access point selection
Before presenting some server code it is important to discuss about access points selection. When some socket calls like connect() are executed, a dialog is presented, allowing users to choose which connection they want to use. To avoid further connection dialogs or even for specifying exactly where you want to put your server, it is interesting to set your access point.
Python for S60 has four special functions for dealing with this issue in the socket module (Python 1.4.x) or btsocket module (Python 1.9.x). From Pyhton for S60 documentation:
- select_access_point(): opens a popup selection where access points are listed and can be selected. Returns selected access point id.
- access_point(apid): creates access point object by given apid. Returns access point object.
- set_default_access_point(ap): sets the default access point that is used when socket is opened. Setting apo to “None” will clear default access point.
- access_points(): lists access points id’s and names that are available.
The basic idea is to create an access point object from an access point ID. Thus, this object is used to set the default access point. For instance:
>>> import btsocket >>> api=btsocket.select_access_point() >>> print api 8 >>> apo = btsocket.access_point(api) >>> btsocket.set_default_access_point(apo)
The access point object has three interesting methods:
>>> dir(apo) ['ip', 'start', 'stop']
With start() it is possible to activate this access point. Run it and after some seconds the phone will show the connection icon at phone home screen. If you are using DHCP, the IP will be negociated in this phase and it may be retrieved with apo.ip(). Finally, the access point connection may be interrupted with apo.stop().
The following code snippet should work with Python 1.4.x and 1.9.x and it will create a access point selection dialog.
# -*- coding: utf-8 -*-
import sys
from appuifw import *
try:
# http://discussion.forum.nokia.com/forum/showthread.php?p=575213
# Try to import 'btsocket' as 'socket' (just for 1.9.x)
sys.modules['socket'] = __import__('btsocket')
except ImportError:
pass
import socket
def sel_access_point():
""" Select the default access point.
Return True if the selection was done or False if not
"""
aps = socket.access_points()
if not aps:
note(u"No access points available","error")
return False
ap_labels = map(lambda x: x['name'], aps)
item = popup_menu(ap_labels,u"Access points:")
if item is None:
return False
apo = socket.access_point(aps[item]['iapid'])
socket.set_default_access_point(apo)
return True
The customized access dialog is below:
Protocols and TCP servers
Everything is about protocols in computer networks, doesn’t it ? In fact they are very important. A protocol is a kind of agreement where rules and roles are defined. They are essential for computer networks, controlling the communication between computer endpoints in all levels (from hardware to applications).
In this post, we will create a simple homebrew protocol to receive files over WiFi network using a mono thread server. Our protocol has the following format:
So, we have three fields, separated by \n:
- An string representing the file name (maximum of 32 characters).
- The size of the file, represented in big endian (most significant byte comes first in memory). When exchanging data between machines with different alignments, endianness matters. While PCs are little endian, machines with MIPS processor are big endian and if you do not pack properly your data it will be misunderstood. Size may be useful for checking if all bytes were received or even if we will have enough space for it.
- File contents.
TCP servers have a specific flow. Basically, you need the following sequence of calls to socket API:
- socket(): it is used to create the socket, our access point to the socket API
- bind(): if you are creating a server, it is necessary to specify exactly where your server will run or your clients won’t find it. Bind does this task, saying to the operating system that we want to create a new server at (IP,PORT). For multi-homed machines (a machine with several IPs), its possible to use 0.0.0.0 to say to operating system (OS) that we want to have our server in all available local IPs (but same port).
- listen(): frequently misunderstood, listen is used to reserve resources in your OS. It tell to OS how many simultaneous connection requests must be handled by it at the same time for this socket. Once the connection is accepted by your server, related resources are released, allowing additional connection requests. listen does not limit the amount of connected clients that you may have, this is done by your application. Which number to put in listen ? The answer is application dependent. If you expects few connections per minute, use an small number, like 5 or 10. Many connections per second ? It better to try to estimate the amount of simultaneous connection requests you may have and use this number in listen.
- accept(): everything is defined, your server is up and running. Time to take a breath and wait for your clients, using accept. Your server code will block when accept is called, waiting for possible incoming connections. When a connection arrives, the program continues its normal execution flow just after accept call. It is possible not to block in a call to accept, using some socket options.
accept() returns two important parameters:
- a socket for describing the incoming connection. Any further communication with the client must be done using this socket.
- client IP address and port, as a tuple.
From this point, you can use recv(), send() to exchange data with your client. Multi thread servers will probably create a new thread, passing the socket to it. This way, it can call accept again and wait for more clients.
Since TCP is a byte oriented stream, you need to wait until a complete name and size arrive, so it is necessary some additional work when receiving. As we will see later, the makefile function may helps a lot.
The server code is below:
# -*- coding: utf-8 -*-
import sys
try:
# http://discussion.forum.nokia.com/forum/showthread.php?p=575213
# Try to import 'btsocket' as 'socket' - ignored on versions < 1.9.x
sys.modules['socket'] = __import__('btsocket')
except ImportError:
pass
from appuifw import *
import socket
import os
import e32
import struct
class FileUpload(object):
""" File upload server class
"""
def __init__(self):
self.lock = e32.Ao_lock()
self.dir = "e:\\file_upload"
if not os.path.isdir(self.dir):
os.makedirs(self.dir)
self.apo = None
self.port = 54321
self.new_line = u"\u2029"
app.title = u"File upload"
app.screen = "normal"
app.menu = [(u"About", self.about)]
self.body = Text()
app.body = self.body
self.lock = e32.Ao_lock()
def recv_file(self,cs,addr):
""" Given a client socket (cs), receive a new file
and save it at self.dir
"""
data = ""
name = ""
size = 0
# waiting for file name
while True:
n = data.find("\n")
if n >= 0:
name = data[:n]
data = data[n+1:]
break
buf = cs.recv(1024)
data += buf
# waiting for file size (may be useful for limits checking)
while True:
n = data.find("\n")
if n >= 0:
# unpack one long (L) using big endian (>) endianness
size = struct.unpack(">L",data[:n])[0]
data = data[n+1:]
break
buf = cs.recv(1024)
data += buf
self.body.add(u"Uploading %s (%d bytes)" % (name,size) + self.new_line)
# waiting for file contents
fname = os.path.join(self.dir,name)
f = open(fname,"wb")
while True:
f.write(data)
data = cs.recv(1024)
if not data:
break
self.body.add(u"Finished." + self.new_line)
cs.close()
f.close()
def server(self,ip,port):
""" Starts a mono thread server at ip, port
"""
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind((ip,port))
s.listen(1)
while True:
(cs,addr) = s.accept()
self.body.add(u"Connect to %s:%d" % (addr[0],addr[1]) + self.new_line)
self.recv_file(cs,addr)
def sel_access_point(self):
""" Select and set the default access point.
Return the access point object if the selection was done or None if not
"""
aps = socket.access_points()
if not aps:
note(u"No access points available","error")
return None
ap_labels = map(lambda x: x['name'], aps)
item = popup_menu(ap_labels,u"Access points:")
if item is None:
return None
apo = socket.access_point(aps[item]['iapid'])
socket.set_default_access_point(apo)
return apo
def about(self):
note(u"File upload by Marcelo Barros (marcelobarrosalmeida@gmail.com)","info")
def run(self):
self.apo = self.sel_access_point()
if self.apo:
self.apo.start()
self.body.add(u"Starting server." + self.new_line)
self.body.add(u"IP = %s" % self.apo.ip() + self.new_line)
self.body.add(u"Port = %d" % self.port + self.new_line)
self.body.add(u"Repository = %s" % (self.dir) + self.new_line)
self.server(self.apo.ip(),self.port)
self.lock.wait()
app.set_tabs( [], None )
app.menu = []
app.body = None
app.set_exit()
if __name__ == "__main__":
app = FileUpload()
app.run()
Finally, this small PC code may be used to send the file. Run it from command line and use the syntax script_name.py server_ip server_port file_to_be_send.
# -*- coding: utf-8 -*-
import struct
import sys
import os
import socket
if len(sys.argv) < 4:
print "%s server_addr server_port file_to_upload" % sys.argv[0]
sys.exit(1)
ip = sys.argv[1]
port = int(sys.argv[2])
full_name = sys.argv[3]
base_name = os.path.basename(full_name)
size = os.path.getsize(sys.argv[3])
print "Sending %s to %s:%d ..." % (base_name,ip,port)
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((ip,port))
f = open(full_name,"rb")
header = "%s\n%s\n" % (base_name,struct.pack(">L",size))
s.sendall(header)
while True:
data = f.read(1024)
if not data:
break
s.sendall(data)
s.close()
f.close()
In the next two pictures we can see these programs working. The command ping was used to test the ad-hoc connection between PC (10.0.0.2) and mobile phone (10.0.0.1), before transferring files.
A possible PyS60 client for sending files to the is below. It is very similar to the PC solution presented, just addeding a PyS60 UI wrapper and a class for selecting files called FileSel.
Take a look in the source code and try to understand how it works. Some socket calls are surrounded by try/catch statements (exceptions) and we will talk about them later.
# -*- coding: utf-8 -*-
import sys
try:
# http://discussion.forum.nokia.com/forum/showthread.php?p=575213
# Try to import 'btsocket' as 'socket' - ignored on versions < 1.9.x
sys.modules['socket'] = __import__('btsocket')
except ImportError:
pass
from appuifw import *
import socket
import os
import e32
import struct
class RxFile(object):
""" RxFile server class
"""
def __init__(self):
self.lock = e32.Ao_lock()
self.dir = "e:\\rxfile"
if not os.path.isdir(self.dir):
os.makedirs(self.dir)
self.apo = None
self.port = 54321
self.new_line = u"\u2029"
app.title = u"RX File"
app.screen = "normal"
app.menu = [(u"About", self.about)]
self.body = Text()
app.body = self.body
self.lock = e32.Ao_lock()
def recv_file(self,cs,addr):
""" Given a client socket (cs), receive a new file
and save it at self.dir
"""
data = ""
name = ""
size = 0
# waiting for file name
while True:
n = data.find("\n")
if n >= 0:
name = data[:n]
data = data[n+1:]
break
try:
buf = cs.recv(1024)
except socket.error:
cs.close()
return
data += buf
# waiting for file size (may be useful for limits checking)
while True:
n = data.find("\n")
if n >= 0:
# unpack one long (L) using big endian (>) endianness
size = struct.unpack(">L",data[:n])[0]
data = data[n+1:]
break
try:
buf = cs.recv(1024)
except socket.error:
cs.close()
return
data += buf
self.body.add(u"Receiving %s (%d bytes)" % (name,size) + self.new_line)
# waiting for file contents
fname = os.path.join(self.dir,name)
f = open(fname,"wb")
n = len(data)
while True:
f.write(data)
n += len(data)
if n % 100 == 0: # a mark at each 100k
self.body.add(u".")
try:
data = cs.recv(1024)
except socket.error:
cs.close()
return
if not data:
break
self.body.add(self.new_line + u"Finished." + self.new_line)
cs.close()
f.close()
def server(self,ip,port):
""" Starts a mono thread server at ip, port
"""
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.bind((ip,port))
except socket.error, (val,msg):
note(u"Error %d: %s" % (val,msg),"info")
return
s.listen(1)
while True:
(cs,addr) = s.accept()
self.body.add(u"Connect to %s:%d" % (addr[0],addr[1]) + self.new_line)
self.recv_file(cs,addr)
def sel_access_point(self):
""" Select and set the default access point.
Return the access point object if the selection was done or None if not
"""
aps = socket.access_points()
if not aps:
note(u"No access points available","error")
return None
ap_labels = map(lambda x: x['name'], aps)
item = popup_menu(ap_labels,u"Access points:")
if item is None:
return None
apo = socket.access_point(aps[item]['iapid'])
socket.set_default_access_point(apo)
return apo
def about(self):
note(u"RX File by Marcelo Barros (marcelobarrosalmeida@gmail.com)","info")
def run(self):
self.apo = self.sel_access_point()
if self.apo:
self.apo.start()
self.body.add(u"Starting server." + self.new_line)
self.body.add(u"IP = %s" % self.apo.ip() + self.new_line)
self.body.add(u"Port = %d" % self.port + self.new_line)
self.body.add(u"Repository = %s" % (self.dir) + self.new_line)
self.server(self.apo.ip(),self.port)
self.lock.wait()
app.set_exit()
if __name__ == "__main__":
app = RxFile()
app.run()
Exception handler and debugging
A new element was presented in the last section: exception handling. It is quite impossible to create a bulletproof programs without any kind of exception handling. In special, the PyS60 socket API does an extensive usage of exceptions to indicate errors like connection timeout, unexpected closed connections and so on. When programming, you need to create some exception handling for almost all socket functions.
Exceptions
For instance, see how to create a simple exception handling when connection to a server. For newer Python versions (1.9.x) you may use settimeout() socket method to limit the connection time. In fact, any socket operation will be timed out using this method (details here).
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.connect(('nokia.com',2222))
except:
print "Can´t connect"
Since port 2222 does not have a server listening on it, the output is:
Can´t connect
But you can obtain more information about the error when using specific socket exceptions. For instance, take a look in the following example, where an exception raised for socket-related errors is used (socket.error):
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.connect(('nokia.com',2222))
except socket.error, e:
print "ERROR: %s" % e
Output (much better then previous one, no?):
ERROR: timed out
Other useful exception is socket.gaierror, related to address resolution problems. In the next example an invalid address is used (nokiacom), generating an exception. And see how we can use several except statements.
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
try:
s.connect(('nokiacom',2222))
except socket.gaierror, e:
print "ADDR ERROR: %s" % e
except socket.error, e:
print "ERROR: %s" % e
Output (more specific exception first):
ADDR ERROR: (11001, 'getaddrinfo failed')
Following the zen of Python statements “explicit is better than implicit” and “errors should never pass silently”, the last example is interesting since it handles all expected exceptions individually. However, the “lazy approach” (first example) may be enough in many situations.
Debugging
Mono thread programs can be easily debugged with exceptions but multi thread programs are a little bit hard to debug. An additional technique is to use log files to debug. Post-mortem analysis are made simple with log files and no much effort is necessary to create your logging strategy.
A log class is suggested in the next code snippet. Just save it in a file called logfile.py and import the object FLOG. Any module may import this object and use it to add log messages with FLOG.add(msg). Caller information is automatically added to the log using the object sys._getframe and a semaphore (created with thread.allocate_lock) is provided to avoid reentrance problems when writing to the file.
# -*- coding: utf-8 -*-
import sys
import time
import thread
__all__ = [ "FLOG" ]
class FileLog(object):
def __init__(self,filename):
self.filename = filename
self.file = open(self.filename,'at')
self.lock = thread.allocate_lock()
def add(self,msg):
# collect caller information from stack call using _getframe
caller = sys._getframe(1).f_code.co_name
line = str(sys._getframe(1).f_code.co_firstlineno)
# create log message
timestamp = time.strftime("[%Y%m%d %H:%M:%S] ",time.localtime())
logmsg = timestamp + caller + ":" + line + " - " + msg + " \n"
# write log message, using a semaphore for controlling the file access
self.lock.acquire()
self.file.write(logmsg)
self.file.flush()
self.lock.release()
FLOG = FileLog("e:\\filelog.txt")
Advanced Socket API
Multicast
Until now we have used only TCP in our examples due to its reliability. In fact, UDP could be used when reliability is not an issue or when you implement it in the application layer. However, UDP is your only solution when talking about multicast and broadcast. In this post we will explore the multicast topic from the programmers perspective (see the related theory in our last post). At this moment, I think it may be more interesting than generic UDP sockets issues (I will explore it later).
Consider the multicast address 224.202.20.30. The following PyS60 program will send a hello world message at each two seconds to this address and in the screenshot you can see the multicast frame, addressed to MAC 01:00:5e:4a:14:1e. Since we are dealing with UDP, the socket is created using SOCK_DGRAM and not SOCK_STREAM.
#
import time
try:
from btsocket import *
except:
from socket import *
from appuifw import note, popup_menu
def sel_access_point():
aps = access_points()
if not aps:
return None
ap_labels = map(lambda x: x['name'],aps)
item = popup_menu(ap_labels, u"Access point:")
if item is None:
return None
apo = access_point(aps[item]['iapid'])
set_default_access_point(apo)
return apo
GROUP = "224.202.20.30"
PORT = 54321
apo = sel_access_point()
if apo is not None:
apo.start()
sock = socket(AF_INET, SOCK_DGRAM)
while True:
n = sock.sendto('hello world',(GROUP,PORT))
print "Message sent (%d bytes)" % n
time.sleep(2)
Instead the known send() function, here it is used sendto(). sendto has an additional parameter, indicating the message destination in the format (address,port). This is necessary since we do not have a phase for connection establishment, with explicit call to connect(), as it is usual in TCP sockets. Thus, client UDP sockets just need to be created and explicitly addressed in sendto(). You can connect an UDP socket, for instance, and use send()/recv() primitives instead non-connected versions sendto()/recvfrom() if you want.
For receiving multicast messages, first create the socket and bind it to the interface. After, it is necessary to join to the desired group, as discussed in our last post. This can be done using two calls to setsockopt() function. The first call enables multicast reception for the interface and the second one tell to the network card which IP we are waiting for. setsockoption has three parameters (there are many socket options and it is better to dedicate a post to the most important ones):
- level:it is used to selected a group of options, like SOL_SOCKET for socket options and SOL_IP for options related to IP layer.
- option name: the value of the option
- option value: new value to the option
inet_aton() converts an IPv4 address to 32-bit packed binary format, as a string four characters (see Python docs).
The last step is to call recvfrom(). Again, recvfrom() is similar to recv() but it will return the address of the incoming package as well (remember: we are not connected).
Using Python 1.9.5 (socket module) it was only possible to send broadcast messages. For receiving, it is necessary to select the access point first but access point functions are only in btsocket and they do not affect socket module. Moreover, btsocket does not have the necessary socket options. So, I did the following PC code for receiving the first ten multicast messages:
import time
import sys
import socket
GROUP = "224.202.20.30"
PORT = 54321
def join_grp(sock,grp,itf):
# enabling multicasting option for itf
sock.setsockopt(socket.SOL_IP,
socket.IP_MULTICAST_IF,
socket.inet_aton(itf))
# joing to multicast group grp in interface itf
sock.setsockopt(socket.SOL_IP,
socket.IP_ADD_MEMBERSHIP,
socket.inet_aton(grp)+socket.inet_aton(itf))
def leave_grp(sock,grp):
# removing
sock.setsockopt(socket.SOL_IP,
socket.IP_DROP_MEMBERSHIP,
socket.inet_aton(grp)+socket.inet_aton('0.0.0.0'))
itf_addr = '10.0.0.101'
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind((itf_addr,PORT))
join_grp(sock,GROUP,itf_addr)
for n in range(11):
(data,addr) = sock.recvfrom(1500)
print "Received ",data,"from",addr
leave_grp(sock,GROUP)
Below are some common problems found in during this test session, just for helping you.
- Windows firewall enabled
- Access point with MAC filter enabled
- Wireshark only working in non-promiscuous mode for my Acer Aspire One
- Missing socket options in PyS60 socket API
- Python 1.9.x seems with better network support than Python 1.4.5
Broadcast
For multicast, a special range of IPs used to send messages for a group of machines. In this case, machines that want to receive message in such group must subscribe to it using socket options. However, it is possible to send messages for all machine in the subnet without any previous subscription using UDP packets and a broadcast address. All machines in the subnet must receive these broadcast messages even when they are not addressed to them. Such messages only will be discarded in the network or transport layer, after checking the package contents.It is not difficult to understand why this kind of addressing was removed in IPv6, doesn’t it ?
There are two four types of broadcast (first two are the most common):
- Limited broadcast: A special broadcast where the destination address is 255.255.255.255. It is called limited because it is never forwarded for other networks by routers, just machines in the local subnet will receive this message.
- Subnet directed broadcast: In this case, the subnet broadcast IP address is used to specify the target subnet. Routers may forward this message but they need to know the subnet mask. For instance, the broadcast address for network 172.16.10.0 with mask 255.255.255.0 is 172.16.10.255.
- Net directed broadcast: The broadcast address is calculated using the default mask for classes A, B or C . So, class A will have broadcast address like <netid>.255.255.255, for instance. Routers will forward these messages but it is possible to disable them.
- All subnets directed broadcast: Quite similar to net directed broadcast but grouping all subnets in this approach. For instance, 172.16.255.255 is the all subnets directed brodcast address for network 172.16.10.0.
In the data link layer broadcast messages use the special MAC address FF:FF:FF:FF:FF:FF, regardless the type of broadcast IP address used, as we can see in the next picture where it was used the subnet broadcast address (10.0.0.255 for network 10.0.0.0/24).
When programming, you do not need any special care when creating broadcast receivers. However, for transmitting broadcast messages, it is necessary to set the socket option SO_BROADCAST to true, in SOL_SOCKET group.
Using Python 1.9.x (socket module) it was only possible to send broadcast messages. For receiving, it is necessary to select the access point first but access point functions are only in btsocket and they do not affect socket module. Moreover, btsocket does not have the necessary socket options.
I will present here the PyS60 version for sending broadcast messages and the PC versions for sending and receiving broadcast messages.
import time
import sys
import socket
from appuifw import note, popup_menu
BCIP = "255.255.255.255"
PORT = 54321
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_BROADCAST, True)
while True:
n = sock.sendto('hello world',(BCIP,PORT))
print "Message sent (%d bytes)" % n
time.sleep(2)
import time
import socket
BCIP = "255.255.255.255"
PORT = 54321
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_BROADCAST, True)
while True:
n = sock.sendto('hello world',(BCIP,PORT))
print "Message sent (%d bytes)" % n
time.sleep(2)
import time
import socket
PORT = 54321
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind(('0.0.0.0',PORT))
while True:
(data,addr) = sock.recvfrom(1500)
print "Received ",data,"from",addr
File like sockets
TCP sockets have flow control but they require some extra coding since we can not precisely control the amount of received bytes when using recv(). However, the socket API in Python provides a very useful method called makefile() that transforms a standard socket object in a file like object. For instance, you can use read() to receive a specific amount of bytes or readline() to receive a complete line (for line oriented streamings, of course). It is important to note that this file like object not is common in other socket API even though it is very interesting and frequently used in Python applications. In this post I want to show some examples using makefile.
In this first example a line oriented client/server is presented. First, take a look in the client. It is used to send five lines to server through the file like object. An special call to flush() is necessary to send any pending byte before slepping (time.sleep() call) and the newline character (\n) is required. The newline is used to control the flow and it is not removed from the streaming at server side.
import socket
import time
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('127.0.0.1',54321))
fsock = sock.makefile()
for i in range(5):
msg = u"Sending line %d ...\n" % i
fsock.write(msg)
print msg
# call flush to send pending bytes
fsock.flush()
time.sleep(1)
sock.close()
The server is really simple as well, as you can see below:
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',54321))
s.listen(1)
while True:
sock,addr = s.accept()
fsock = sock.makefile()
print "New connection from",addr
while True:
line = fsock.readline()
if not line:
print "Connection closed"
break
print "=> ",line
A better and more interesting example may be created when we need to decode headers or specific protocol issues. Suppose we have created a protocol where the first two integers (each one with four bytes in big endian format) are version and size, respectively. After them, the data is transmitted. See how this could be done using makefile (client and server are below):
import socket
import struct
import random
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect(('127.0.0.1',54321))
fsock = sock.makefile()
# creating a random header
version = random.randint(0,5)
size = random.randint(0,10)
header = struct.pack('>LL',version,size)
print "Version:",version
print "Size:",size
fsock.write(header)
data = 'X'*size
print "Data:",data
fsock.write(data)
fsock.flush()
sock.close()
import socket
import struct
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind(('127.0.0.1',54321))
s.listen(1)
while True:
sock,addr = s.accept()
fsock = sock.makefile()
print "New connection from",addr
# decoding header
header = fsock.read(8)
(version,size) = struct.unpack('>LL',header)
print "Version:",version
print "Size:",size
data = fsock.read(size)
print "Data:",data
sock.close()
Further reading
The following books use C as programming language to explain the socket API (The behaviour of Python sockets is similar because the C socket API has become a de facto standard across most programming languages).
- Beej's Guide to network programming. Beej describes the socket API, in his own words, "with hopefully as little hassle as humanly possible". Concise and straight to the point, this guide remains a great reference.
- TCP/IP Illustrated, Volume 1, 1994, by W. Richard Stevens. Despite being quite old, this is still a fantastic book. It covers in details several important protocols, including: TCP, UDP, IP, ICMP and others.
- TCP/IP Illustrated, Volume 2 This provides implementation information and source code for BSD 4.4, the starting point for many network stack implementations.
- UNIX Network Programming, third edition, 2004, by W. Richard Stevens, Bill Fenner and Andrew M.Rudoff. This book was originally written by Stevens but after he passed away, in 1999, the was released in a new edition with new co-authors. This discusses both the socket API, and more recent protocols, like IPv6 and SCTP. It is a book for programmers and a great reference.
Several books have been written specifically for Python developers:
- Foundations of Python Network Programming, 2004, by John Goerzen. This covers not only the socket API but also several important modules found in Python.
Conclusions
Sign in to comment…
Jumpjack said...
Hamishwillee said…
Feedback on document: (still more to come)
- @Jumpjack raised a comment "how can I set the socket timeout? Is it possibile to have an exception raised if phone can't connect after xx seconds?" Is this addressed?
- I really like your introduction, but it reads like a forward to a book, not to a chapter. I've moved the references to a "Further reading" section and made it just a little less personal (sorry, I really do like it, but its very different to the rest of the book which is to the detriment of readability). For the beginning of most chapters we have
- a very short section just explaining who the information is for. The introduction then contains the sales pitch/what we cover in the rest of the text.
- In the introduction I think its worth talking about what we'll get in the Advanced Networking section.
- We do need to cover how Python and PyS60 networking differs, so I've tried to capture that in the introduction. Is it sufficient?
- Section Basic Network Programming (Python on Symbian)#Access point selection has note that it needs to be updated for PyS60 2.0 - which has now arrived. There is another couple of notes like this. When do you plan to update?
- Need to use consistent mechanisms for linking back to TOC. Still under discussion
- This is document on Python 2.0. I think we should add single note or section that captures the differences "as an aside", but otherwise remove from the text.
- Can you make the example code available as tested/known to run projects? Something that people can simply run/build rather than having to copy text from scripts
- We markup inline code using {{Icode| the-inline-object}}. If its a function, we use () in name to indicate this.
- This needs a conclusion. I've added the heading, but can you fill it in?
- For images we tend to use a left aligned thumbnails and a caption: [[File:name|thumb|none|300px|This is my caption]]
- Do broadcast and multicast acutally work? I know there have been issues with the C++ API in the past.
It also occurs to me that perhaps the sections on debugging should be in a generic debugging section. Minimally these should cross reference the basic python concepts section.
Hope to continue this tomorrow.
--Hamishwillee 07:28, 15 March 2010 (UTC)
















how can I set the socket timeout? Is it possibile to have an exception raised if phone can't connect after xx seconds?
--Jumpjack 13:03, 19 September 2009 (BST)