Message-ID: <41ABE8EA.5020404@zacbowling.com> Disposition-Notification-To: Zac Bowling Date: Mon, 29 Nov 2004 21:28:42 -0600 From: Zac Bowling User-Agent: Mozilla Thunderbird 0.9 (Windows/20041103) X-Accept-Language: en-us, en MIME-Version: 1.0 To: gaim-devel@lists.sf.net Subject: Adding UPnP Support Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I know this is a very long email so bear with me. None of this is very hard for most programmers. As a lot of you may know, I've been working on adding support for UPnP into Gaim. As of yet, I have not written any code into Gaim but I have made many working example apps and they all work. I'm trying to make gaim automaticly setup port forwarding with any UPnP capable router, so when you and your buddy are both behind a different NATs you can still do your file transfers, direct connects, and whatever else you can think of. I just don't believe that I'm going to have much time time personally to do any coding since my vaction time is over and I've got to much work to do. So, I have documented everything I have learned and written a very basic, step by step cheat sheet for any developer to pick up where I left off. I would love to see this get written as a patch for Gaim but it would work great as a plug-in also. First off, lots of routers have support for UPnP IGD now like most consumer size routers and firewalls (like most Linksys, DLink, Netgear, 2Wire, etc...), Internet Connection Sharing in Windows XP, and any Linfux router running UPnP IGD for Linux (http://upnp-igd.sf.net). The Windows clients for AIM, MSN, and Yahoo already take advantage of this. I'm not trying to add full UPnP support so its not as complicated as some would think. In Windows platforms, you can make a call to a COM object or an API and Windows will do all the work for you, but for Linux its a bit more complicated becuase you have to go down to the protocol itself to make the calls. There has been work to impliement the server side of the Internet Gateway Device (IGD) spec under Linux (http://linux-igd.sf.net), but the client side libraries to take advantage of UPnP in linux are lame and way over-bloated to be useful (http://upnp.sf.net). Everyone doing it in Linux apears to have used the tool created by Intel that generates a bunch of C so you can just drop it in and go. Unfortunatly their code creates its own web server, its own XML/SOAP handling framework, and a gaint over bloated set of mostly unused APIs that don't do much more then be huge (most of this is for the event framework of SOAP but we don't need to impliment any type of callbacks). I personally believe the UPnP protocol is just another fad and Microsoft promotional protocol. It's way over bloated and does a lot of other stupid functions that serve no pratical purpose, like trying to connect your TV media players to your clocks to your toothbrushes and to whatever else has the ablity to be in wired in together with it. However it does have one great part. They have the first standarization of any kind between these types of devices to be able to preform some of the basic functions provided by most of these devices. Like for example, you can call simple things like setting up port forwarding for NAT transversal, querying your external ip of your router, and getting a call back event when your connection between your provider and your router goes down, all from inside your code. I've done all the research on all the protocol documentation, analyzed and documented packets used by other UPnP programs like MSN and AIM, and I was able to reproduce working examples that work with 3 different UPnP routers (2 different Linksys models and 1 Netgear) and get it to work on both linux and windows. I believe that it doesn't need more then a few hundred lines of code at the very most and about 90% of that should be just string playing and defining a bunch of very long but static multiline strings for the funny HTTP headers and all the XML/SOAP bodies for the messages. On start gaim will need to: * broadcast an http-like message over UDP to discover your UPnP devices. * listen for a possible http-like response from your router over that same UDP port if any UPnP devices exist. * make a simple http request to URL in the location tag in that last UDP message, and pull a URL for the specific function you want out of the returned XML. When your ready to open the port just do a simple little http post to that URL form before for each port you want to open. Your router will setup up port fowarding for you automaticly. The entire procedure requires no configuration or interaction by the user to use UPnP, so maybe we don't have to hear as many complaints by lusers about their file transfers not working behind their firewalls. The only added strain on anything is that when Gaim is ran inside a private IP range or in the default configuration address range, it will send a single UDP broadcast out (maybe two since its UDP) when it starts. It will then have to listen (only maybe for 1 minute or so) for a possible response that could come back from your UPnP devices. The only security risks I can think of is that you end up listening for a UDP packet comming from an unknown source and even though its hard to see any issues with this, if left unchecked someone could spoof they are a UPnP device and have you download from them. Someone asked me what UPnP IGD port fowarding does when mulitple applications request to use the same port. Just to anwser people's questions on that one thing, UPnP IGD devices usally just overwrite an existing port mapping with the new one. It also allows you to do IP filtered port forwarding, meaning you can optionaly pass the IP address of the device that you know will be contacting you. So if you want to transfer a file with your friend, and your IM client knows your friends IP, but your mom is in the other room is doing a bunch of file transfers with some of her friends on the same port with MSN messanger, you can both still connect to each other. I don't know about the specifics on getting this to work with the remote IP specific version of this with each protocol because you have to be able to get the IP public address of the other person before he connects to you and some protocols may not have this ablity, but if they do you have already got AIM and MSN beat on that feature without trying. I should mention also that UPnP IGD allows you to set a timeout on the port mapping but I haven't seen applications even use it. They have all just been passing 0, so their port mappings never expire (even though the WRT54G from Linksys will automaticly expire mappings set for 0 in 24 hours because it doesn't have support for a maximum number of mappings like most devices do). The details on how to impliment UPnP can be found below: -------------------- First off, you should always send 2 UDP broadcasts to 239.255.255.250:1900 in this http like format. (in C String Syntax just because I'm a nice guy) "M-SEARCH * HTTP/1.1\r\n" "Host:239.255.255.250:1900\r\n" "ST:upnp:rootdevice\r\n" "Man:\"ssdp:discover\"\r\n" "MX:3\r\n" "\r\n" "\r\n" XP sends this when it joins a network: M-SEARCH * HTTP/1.1 Host:239.255.255.250:1900 ST:upnp:rootdevice Man:"ssdp:discover" MX:3 Its a good idea to send two a few seconds apart becuase UDP isn't a nice protocol for making sure everything has found its final destination... You should recieve a bunch of udp responses from all the UPnP devices on your network so you should maybe have a thread and listening socket for any response you might get later. The responses will come back to on the same UDP port used to sendout the first UDP query. Its a good idea maybe to keep that thread open because it may be useful later when you want to call your port forwarding requests unless you plan on sending it only right before known communications on that port are just about to start. The response message you want most from all the devices, should look something like this (from my Linksys BEFSR41 firmware v1.05.00): NOTIFY * HTTP/1.1 HOST:239.255.255.250:1900 Cache-Control:max-age=120 Location:http://192.168.1.1:5678/rootDesc.xml NT:urn:schemas-upnp-org:service:WANIPConnection:1 USN:uuid:upnp-WANConnectionDevice-1_0-0090a2777777::urn:schemas-upnp-org:service:WANIPConnection:1 NTS:ssdp:alive Server:NT/5.0 UPnP/1.0 or like this (from my Linksys WRT54G with my own custom firmware based on version v2.0.2) HTTP/1.1 200 OK ST:urn:schemas-upnp-org:service:WANIPConnection:1 USN:uuid:000625d8-095f-0006-25d8-095f0232aa18::urn:schemas-upnp-org:service:WANIPConnection:1 Location: http://192.168.1.1:5431/dyndev/uuid:000625d8-095f-0006-25d8-095f0032aa18 Server: Custom/1.0 UPnP/1.0 Proc/Ver EXT: Cache-Control:max-age=1800 DATE: Mon, 29 Nov 2004 16:58:01 GMT The part of the message that is the most valuable is the last part of the USN value just after the "urn" tag. You can use that to filter out the specific device we want to talk to. We want a device that follows the Internet Gateway Device (IGD) specification in the UPnP spec, so you want to look for a value that says "schemas-upnp-org:service:WANIPConnection:1". You will then need to make a basic HTTP request back to the value that was passed to you as the "Location" tag to get some more information. It should be as simple as this: GET HTTP/1.1 HOST: : ACCEPT-LANGUAGE: en You can also pass a referer or an acceptable language tag but I don't believe its even nessisary. You should recieve an XML discription with what services are provided by your device it should look something like this: 1 0 http://192.168.1.1:5678 urn:schemas-upnp-org:device:InternetGatewayDevice:1 /index.htm Linksys BEFSR41/BEFSR11/BEFSRU31 Linksys Inc. http://www.linksys.com Internet Access Server Linksys BEFSR41/BEFSR11/BEFSRU31 uuid:upnp-InternetGatewayDevice-1_0-0090a2777777 00000-00001 ... On some devices, they might provide a lot of functions in this xml file so sometimes it can be quite long sometimes spaning multiple packets so a basic http function is good. You are looking for a set of tags looking something like this: urn:schemas-upnp-org:service:WANIPConnection:1 urn:upnp-org:serviceId:WANIPConnection http://192.168.1.1:2468//WANIPConnection /WANIPConnection /WANIPCn.xml or like this on my WRT54G: urn:schemas-upnp-org:service:WANIPConnection:1 urn:upnp-org:serviceId:WANIPConn1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 /dynsvc/WANIPConnection:1.xml More specificly, you are looking for a "ServiceType" tag that equals "urn:schemas-upnp-org:service:WANIPConnection:1". After you have that, you want to remember the controlURL tag value for later until you are ready to fire your port forwarding actions. The specificiation for UPnP says "To provide firewalls and proxies greater administrative flexibility, SOAP specifies that requests must first be attempted without the MAN header or M- prefix. If the request is rejected with a response of "405 Method Not Allowed", then a second request must be sent using the MAN header and M-prefix. If that request is rejected with a response of "501 Not Implemented" or "510 Not Extended", the request fails. (Other HTTP responses should be processed according to the HTTP specification.)" For the attempt we will push the this http post request to the router: POST HTTP/1.1 HOST: : CONTENT-LENGTH: CONTENT-TYPE: text/xml; charset="utf-8" SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping" or if the that returns any error like stated in the spec then make the request with a message like this: M-POST HTTP/1.1 HOST: : CONTENT-LENGTH: CONTENT-TYPE: text/xml; charset="utf-8" MAN: "http://schemas.xmlsoap.org/soap/envelope/"; ns=01 01-SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping" The body of both requests should be a basic SOAP message in the following format: #string# #ui2# [TCP|UDP] #ui2# #boolean# #string# #ui4# Replace these values with your own: #ui2# = 2 byte unsigned integer #ui4# = 4 byte unsigned integer #boolean# = boolean #string# = string [TCP|UDP] = a "TCP" or "UDP" string Real examples of it working are below: You may get an error response back if something happens but I'm unsure what they all look like. The documentation says it should always return a XML SOAP type error but i've seen also sorts of information come back durning an error. For some apps, the goal is "if it works, it works, and if it doesn't, doesn't". I'm guessing that will be the general feeling with the gaim implimentation since its not a requirement of gaim to work, just a nice feature to add support for people with UPnP IGD devices and are behind a NAT. That I believe should be the minimum required to allow almost any application to use the UPnP IGD port forwarding system. This specific technique works on my system and I've seen it done by different software like some closed source FTP programs, file share apps, and even Skype for both Windows and Linux. AIM, Yahoo and MSN all call the Windows API to handle it all, but with a little work we can do them one up. See more on the UPnP spec at http://www.upnp.org/download/UPnPDA10_20000613.htm I really hope someone will finish what I have started. I will be here to help if anyone needs anything or has any questions. I'm always check my email about every hour and I can work on this late at night or on the weekends. I have lots of more documents and packet dumps but I put all the relevant information below. I hope that helps anyone with any questions! Zac Bowling zac@zacbowling.com Here is a sample of XP getting the discription XML from my WRT54G: ------ LOCAL ------ GET /dyndev/uuid:000625d8-095f-0006-25d8-095f0032aa18 HTTP/1.1 HOST: 192.168.1.1:5431 ACCEPT-LANGUAGE: en ------ WRT54G ------ HTTP/1.0 200 OK SERVER: LINUX/2.4 UPnP/1.0 BRCM400/1.0 DATE: Mon, 29 Nov 2004 16:58:01 GMT CONTENT-TYPE: application/octet-stream Cache-Control: max-age=1 PRAGMA: no-cache Connection: Close 1 0 urn:schemas-upnp-org:device:InternetGatewayDevice:1 Residential Gateway Device Linksys Inc. http://www.linksys.com/ Internet Access Server WRT54G v2.02.2 http://www.linksys.com/ uuid:000625d8-095f-0006-25d8-095f0032aa18 urn:schemas-upnp-org:service:Layer3Forwarding:1 urn:upnp-org:serviceId:Layer3Forwarding:11 /uuid:000625d8-095f-0006-25d8-095f0032aa18/Layer3Forwarding:1 /uuid:000625d8-095f-0006-25d8-095f0032aa18/Layer3Forwarding:1 /dynsvc/Layer3Forwarding:1.xml urn:schemas-upnp-org:device:WANDevice:1 urn:schemas-upnp-org:device:WANDevice:1 Linksys Inc. http://www.linksys.com/ Internet Access Server WRT54G v2.02.2 http://www.linksys.com/ uuid:000625d8-095f-0006-25d8-095f0132aa18 urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1 urn:upnp-org:serviceId:WANCommonIFC1 /uuid:000625d8-095f-0006-25d8-095f0132aa18/WANCommonInterfaceConfig:1 /uuid:000625d8-095f-0006-25d8-095f0132aa18/WANCommonInterfaceConfig:1 /dynsvc/WANCommonInterfaceConfig:1.xml urn:schemas-upnp-org:device:WANConnectionDevice:1 urn:schemas-upnp-org:device:WANConnectionDevice:1 Linksys Inc. http://www.linksys.com/ Internet Access Server WRT54G v2.02.2 http://www.linksys.com/ uuid:000625d8-095f-0006-25d8-095f0232aa18 urn:schemas-upnp-org:service:WANIPConnection:1 urn:upnp-org:serviceId:WANIPConn1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 /dynsvc/WANIPConnection:1.xml urn:schemas-upnp-org:service:WANPPPConnection:1 urn:upnp-org:serviceId:WANPPPConn1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANPPPConnection:1 /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANPPPConnection:1 /dynsvc/WANPPPConnection:1.xml ------ ------ ------ Here is a sample of XP setting up Port Forward: ------ LOCAL ------ POST /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 HTTP/1.1 HOST: 192.168.1.1:5431 CONTENT-LENGTH: 696 CONTENT-TYPE: text/xml; charset="utf-8" SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping" 27360 UDP 8615 192.168.1.101 1 msmsgs (192.168.1.101:8615) 27360 UDP 0 ------ WRT54G ------ HTTP/1.1 200 OK DATE: Mon, 29 Nov 2004 16:58:03 GMT Connection: Keep-Alive Server: LINUX/2.4 UPnP/1.0 BRCM400/1.0 Content-Length: 289 Content-Type: text/xml; charset="utf-8" EXT: ------ ------ ------ Here is another sample of XP deleting a port mapping (which souldn't be required but its nice to document this): ------ LOCAL ------ POST /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 HTTP/1.1 HOST: 192.168.1.1:5431 CONTENT-LENGTH: 415 CONTENT-TYPE: text/xml; charset="utf-8" SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#DeletePortMapping" 41209 TCP ------ WRT54G ------ HTTP/1.1 200 OK DATE: Mon, 29 Nov 2004 16:58:27 GMT Connection: Keep-Alive Server: LINUX/2.4 UPnP/1.0 BRCM400/1.0 Content-Length: 295 Content-Type: text/xml; charset="utf-8" EXT: ------ ------ ------ Here is another sample of XP unsucessfully getting the external IP (since I'm using PPP/PPPoE you have to query that space to get the IP adderess): ------ LOCAL ------ POST /uuid:000625d8-095f-0006-25d8-095f0232aa18/WANIPConnection:1 HTTP/1.1 HOST: 192.168.1.1:5431 CONTENT-LENGTH: 296 CONTENT-TYPE: text/xml; charset="utf-8" SOAPACTION: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress" ------ WRT54G ------ HTTP/1.1 200 OK DATE: Mon, 29 Nov 2004 16:58:01 GMT Connection: Keep-Alive Server: LINUX/2.4 UPnP/1.0 BRCM400/1.0 Content-Length: 353 Content-Type: text/xml; charset="utf-8" EXT: 0.0.0.0 ------ ------ ------