Ethernet

[rimai]

Yeah, looks like you are not pushing data to the portal.

[rimai]

The wireshark activity shows me the module sending 4 or 5 bytes at a time.
We need to optimize this to at least 800 byte chuncks. Is there any setting?

[DrewPalmer04]

My cable comes in today! Yay!!!

[00Warpig00]

The wireshark activity shows me the module sending 4 or 5 bytes at a time.
We need to optimize this to at least 800 byte chuncks. Is there any setting?

I knew you were gonna say that... I was looking at the packets when I sent that capture session to you and I was thinking the 20 or so byte header was 5 times longer than the data in each packet. No wonder the SAX parser errors out since you mentioned the SAX parser seems to be picky about packet sizes.

So I dug into the Wiznet manual some over the weekend and discovered there are some settings that can be set to change when and how serial data was sent over to the ethernet side. It appears as the default for the Wiznet module is to just throw data from serial to ethernet interface as fast as possible and this seems to cause the data portion of ethernet packets to remain small and be inconsistent. It is creating small Ethernet packets on the fly just throwing them at the wall as more data is still coming in from the serial side. The Options tab on the Wiznet config utility gives you some options to control the creation of ethernet packets as well as how long to stay connected.


From the Wiznet manual

Inactivity time

After the connection is established, if there is not data transmission within the time defined
in Inactivity time, the connection is closed automatically.
The default value is ‘0’. If ‘0’ is set, this function is not activated. In this default setting, the
connection is maintained even though there is no data transmission. In order to close the
connection, the ‘Close’ command should be given.
This function can be used for more than two or more systems to connect to a WIZ105SR
module. If one system holds the connection to the WIZ105SR, other systems can not connect
to the module. If there is no data transmission during the time defined in Inactivity time, the
connection will be closed for other system to be connected.
Inactivity Time also can be useful for the case when the server system is unexpectedly shut
down. In this status, if there is not any data communication during the time defined in the Inactivity time, WIZ105SR will close the connection and enter into waiting status.

I changed the "inactivity time" to 15 seconds so if there is no data transmitted for 15 seconds the socket connection will be closed. I have not done any packet sniffing yet since I made this change to see if the 15 second timeout is ever reached and and whether the session disconnects or not but I figured for starters 15 seconds of no activity would be better and force closed the socket without having to send a "CLOSE" command as the manual states would be required. Also I thought there could be a chance that the default setting of "0" or "stay connected" could be problematic since the RA needs to communicate both with my Android app and the portal. So for now it's at 15 seconds but this may change. Since I need to sniff the packet activity of the of the actual Wiznet module and whatever it may be connected to I may have to find a hub somewhere to connect the Wiznet module to in order to snoop on that traffic without causing traffic to the module myself. Hoping I don't have to go that far and can get everything working by other means.

The next setting is for "data packing" and the config utility gives you three options.

From the Wiznet manual

Data Packing Condition

You can designate how the serial data can be packed to send to the Ethernet.
In default mode, WIZ105SR checks the serial buffer and sends any received data to Ethernet
without any standard. Therefore, the data can be regarded as error data due to inconsistent
data lenth. In order to solve this problem, the data from serial can be packed according to
the specified condition before sending to Ethernet. The condition is as below.

Time: If there is not any new data during designated time, all data in the serial buffer is
converted to Ethernet packet. If new data is received, the time is re-calculated and
the received data is saved in the serial buffer. Time condition can be utilized when the
serial data size is variable and has no ending character, but needs to be delivered in
frame unit. (If the value is ‘0’, option is not used)

Size: When data is collected as much as the defined size in the serial buffer, it is transmitted
to Ethernet (If the value is ‘0’, option is not used)

Character: The data is collected in the serial buffer until specified character is received, and
transmitted to Ethernet. The character can be specified in Hex formation. (The value is
‘0’, this option is not used)
One of three conditions is satisfied, the data is transmitted to the Ethernet.
Ex) Delimiter: Size=10, Char=0x0D
Serial data : 0123456789abc
Ethernet data : 0123456789
“abc” data remains in the serial buffer of module.

The config utility suggests you set packet sizes from 0~255 bytes
I tried 255 and this didn't seem to resolve any issues. I also keyed in 1024 even though it exceeded the ~255 byte setting the config utility displays. The config utility allowed the setting and this allowed me to make an occasional connection and read data with the Android app but was still giving SAX Parser errors for the bulk of the attempts in my case. It was like I would get lucky once in a while and it would parse properly.
After looking at the definition of the "TIME" function in the Wiznet manual. It seemed this would be a good setting to play with as it didn't seem to define a set packet size but instead reads as if it will just wait until time expires and create a packet out of the entire contents of the serial buffer. This seemed like a great idea to try. So I did and I set the time to wait to 3000ms (3 seconds) figuring that was plenty of time for the RA to send all the data it wanted to into the serial buffer at 57600 baud. This setting seems to have fixed the SAX Parser errors in whole with the Android app. I still get an occasional error when the app tries to hit the controller by itself at the 15 minute intervals I have set on my Android app but If I tell the android app to query the controller by hitting the refresh button I NEVER receive a SAX Parser error. Looks like the Android app portion of the dragon may be slain.

I still do not appear to be pushing data to the portal properly and I am not sure why, but I believe I may just not fully understand how the RA and the portal communicate with each other in general. When I log into the portal two day old data is showing along with my banner showing two day old data. Then 30 seconds later the update icon flashes and current data is displayed. 30 seconds later the update icon flashes again and current data is showing again. I have experimented with different IO module ports being open and shorted and the portal dashboard when it updates every 30 seconds or so IS showing me current LIVE data, But this data does not seem to be getting saved to the portal database. I am guessing at this point that this 30 second interval where the update icon flashes and current data is displayed is actually the portal querying the RA at this point strictly for dashboard display and not the RA pushing data to the portal database.

For now here are all my settings for the Wiznet module as it sits and everything appears to work except the portal is not recording data being sent from my RA. So it appears the data is not making it from the RA to the portal for the purposes of the data being saved in the database but it is having successful two way communications or I would not have complete control of my RA+ via the portal and the portal dashboard would NEVER show current data if the RA wasn't sending it. Am I misunderstanding something here??

Here is my current Wiznet Module configuration. Any suggestions on changes that may make the portal start to save data from the Wiznet module?







Nick

[lnevo]

This might help..

http://forum.e-lab.de/topic.php?p=8780

[rimai]

Cool!!!
Great stuff!!
Yes, I think the time is the one to use, but 3000ms may be too much....
The RA sends data at 57600 baud, 1 start bit, 1 stop bit and 8 data bits. This means 10bits for each character. So, RA is sending 5760 characters per second. I think 200ms would be enough to distinguish a break in sending serial data.
Did you set the Client side of the wiznet?
What you have is the Server side. It's when you initiate the connection.
The data logging of the Portal is the Client side.
RA initiates the connection and send data to the Portal every 5 minutes.
You must configure the remote host ip and port, which is www.reefangel.com [198.171.134.6] on port 80.
You are correct to think that live data does not get logged. It's just live data being displayed.

[00Warpig00]

looking at my config on the network tab above this is the Server IP and port boxes I have filled in with the IP 198.171.134.6 and port 80. Is this incorrect? should I not be using "mixed mode" as is checked on the network tab? I have not played with "Client Operation Mode" check mark on the network tab at all.

Nick

[rimai]

That should be it...
Mixed mode with the ip and port you posted above.
Every 5 minutes, RA is supposed to initiate a connection to the portal.
Can you try to sniff that?

[00Warpig00]

Hmm my data from two days ago was coming through from the WiFi mod so I know that my RA+ is sending the serial data to the module since I have made no code changes.

If the Android app is 100% working, and it is... this means that the level converter is receiving 5V serial data from the RA and converting the level down to 3.3V and passing the serial data to the Wiznet module successfully and the 3V serial data is being read properly by the RA since I can switch my relays on/off with the Android app. The Wiznet module is converting Serial to Ethernet and back and sending packets correctly or the Android app would never receive an update otherwise and I get updated info (when I ask for it) but the portal is just not getting it when the RA/Wiznet just send it without request... I have this right? This eliminates a hardware wiring issue\level converter issue no? So it has to be something in the Server\Client\Mixed mode of the Wiznet module. the part you were worried from the beginning may be the hard part... I admit my TCP/IP Switching\Routing may be a bit rusty and the last time I needed to spy on Ethernet interface A sending data to Interface B from computer interface C I needed a hub thrown in the mix with the Interfaces A and C both connected to that hub since switches send traffic only to the specific mac address's port and not to all ports. Hmm I dont have a hub around or an advanced enough switch that would allow me to send that traffic to two ports. Could maybe spoof the mac address of the Wiznet module with my PC and listen in that way... I may need to freshen up on a few things...
My router should not be stopping outbound traffic from the Wiznet module... I was going to ask if you can see a connection to the database server from my public IP but I know you can or i would not be able to type this post. Problem there is unless you sniff all inbound traffic from my public IP you have no idea whether the traffic is from my PC or from my RA since they both come in on port 80...

One more thing. I put the Wiznet mod into Client only mode for the past 20 minutes or so and i still do not see any updates to my banner. so the problem seems to exist whether I am in mixed mode or client mode. Also Client mode breaks the Android app with a timeout (like I expected it would) so the Wiznet module is set in Client mode.

Nick

[rimai]

If you are all in the same subnet and you are all connected through hard wire, you should be able to sniff it without doing anything. You must have your pc hard wired though.
If your pc is wifi, the simpler way is to change the remote host ip address to your internal computer ip address. I'm sure wireshark should see that traffic coming in, since it is targeted to your ip.
You may need to implement a webserver while sniffing, just so the firewall doesn't drop the packets.

[00Warpig00]

If you are all in the same subnet and you are all connected through hard wire, you should be able to sniff it without doing anything. You must have your pc hard wired though.
If your pc is wifi, the simpler way is to change the remote host ip address to your internal computer ip address. I'm sure wireshark should see that traffic coming in, since it is targeted to your ip.
You may need to implement a webserver while sniffing, just so the firewall doesn't drop the packets.

There I go over thinking things... lol you are correct I don't really need to spy on the connection between RA and your server I just need to see if RA is sending data out every 5 minutes via the Wiznet module. I can do this by targeting the IP address in my Wiznet mod at my computer instead of at your server... If I did this with a twist cable from interface to interface would be the best method...

Nick

[00Warpig00]

Roberto,

check your PM's I sent you another capture file. Looks like the module is attempting to reach out at least to my computer it was.

Nick

[rimai]

I didn't see any response back from your computer to the module. Was the firewall just dropping the packets?
Can you open tcp 80 and run a webserver to see the response back?

[00Warpig00]

New PM for you. i thought the firewal was open. lol

I never saw any ACK's either just the SYN's

Nick

[rimai]

Yes, I do, but the capture doesn't show even a connection established

[00Warpig00]

hmm same webserver in fact you are getting the ZIP's from I am sending you. I could hook it all back up how it was and totally turn off my Windows firewall but I'm not sure it would matter port 80 appears to be open.

I thought the no response was strange too. maybe something wrong with the Wiznet module... since I freaked it out and reset it and got it working again the link and data lights on the module are not lighting up anymore even though the module seems to be working. Wonder if I really fried something. I have another should be here Wed or Thurs. Was hoping it would be delivered today it shows sitting at the UPS depot in the next town over since the 29th. But not scheduled for delivery according to their website until Thursday. Typical UPS let it sit there for 5 days.
Lol


Nick

[00Warpig00]

Ok new Wireshark session in you PM's

looks like while testing pointed at my local webserver I have something in my Firewall settings that was causing the Wiznet packets to be dropped. this time firewall was off. Wiznet mod was in Mixed mode. Wiznet module appeared to take several minutes to get serial data from RA and when it did made connection to my local webserver. You will find Packets 297 and 299 particularly interesting I think.

Thinking about putting the Wiznet Module in my DMZ for some testing to your server. Maybe I am having some strangeness from my router dropping the Wiznet traffic in similar fashion to my Windows firewall.

Nick

[rimai]

Great!!!
So, we know the module is requesting pages... Point it back to reefangel.com server and let's see what it does.

[00Warpig00]

Great!!!
So, we know the module is requesting pages... Point it back to reefangel.com server and let's see what it does.

Yeah... It has been pointed there all night long. Also in my DMZ. does not appear to be any change in behavior. Not sure if those packets are reaching your server. Don't know why they wouldn't be, I seem to have no other issues with any of your websites features, and I surely didn't have this problem with the Wifi mod. so again really shouldn't be a firewall issue on my end or the wifi mod would behave the same. While looking at the packets did the data appear to be what your webserver is looking for? Malformed data possibly? is your webserver ever ACK'ing my SYN's from my Ethernet module? May be hard to tell since I have been hitting your webserver kinda hard lately with my browser and everything is on port 80. Kinda like looking for a needle in warehouse full of needles.

If you want to set up a schedule where I wont hit your server with my web browser (a web browser blackout if you will) and allow only the Wiznet mod to communicate I can stay off your site for a few hours to make it easier to inspect packets from my public IP.

Nick

[rimai]

You should still be able to sniff your LAN and capture the wiznet traffic from your computer.
I do see your webserver returning a HTTP 404 error, which is file not found. This indicates the wiznet did reach your server and your server did return something to the module.
I just don't know why it can't reach the reefangel.com server.
You can also connect the wiznet to the usb-ttl cable and send HTTP request using a serial terminal like putty or tera term or even serial monitor from arduino to see what kind of response you are getting from the server.
I'd try sniffing LAN first though.

[DrewPalmer04]

nvm...figured it out with the serial monitor

[DrewPalmer04]

any update on your side of the ethernet add-on?

[rimai]

No time. I'll try over the weekend

[DrewPalmer04]

Oh sorry. Thanks for doing that one mine. But I was curious about warpig's update.

[00Warpig00]

Nothing yet. Got my second Wiznet 105SR today as well as a Wiznet 110SR. Going to try the 2nd 105 since it is a quick unplug/plug replacement and config with the config utility first just to see what happens. I haven't had the time yet to do any sniffing of the Wiznet module on my local LAN trying to communicate with Roberto's server. Trying to find a hub laying around so I can use an wireshark connection(c) to sniff traffic between the wiznet module(a) and Roberto's server(b) on both my LAN and my WAN port on my router. I'm having a really hard time believing outbound port 80 traffic specifically (only) from the wiznet module is being dropped during transit over the internet between my WAN port and Roberto's server. This to me means either the traffic is never making it out my WAN port or Roberto's server is dropping it for some reason. I'd like to more than guess which it may be.

Nick

[DrewPalmer04]

Mine is corrupting the serial data from the string. Not sure why. I've been racking my brain for days about why.

[00Warpig00]

I don't believe the Wiznet 105SR is corrupting the XML string. I haven't really taken a close look at the string that is sent out. I can say that if the string sent out when RA initiates the transaction is identical to the string when the portal OR the Android app initiates the connection there should be no problem with the data. My problem appears to ONLY be when RA sends data at it's 5 minute intervals. ALL other two way communication seems to work flawlessly whether it be via the portal or by the Android app. If the request comes into the RA (Server Mode) it's all good if RA just says hello on it's own (Client Mode) the Portal never shows any data saved.

Nick

[00Warpig00]

SUCCESS!!!! The Wiznet 105SR is fully functional at this time!! now for some reliability testing over the next few weeks. Turns out Whatever I did to my first 105SR module (The I probably shouldn't try this but...) idea I had last week that caused the first module to wig out does appear to have damaged the module somehow. the only other thought I have is that MAYBE the pullup resistor is not strong enough on the reset line and some strange behavior (that the manual warned about) MAY be resulting. Either way the same cable I was using for the first module is plugged right into the second module (pullup resistor is the same resistor) and in this module it works. I MAY try a stronger pull up resistor but I am betting I fried something on the first module with my EXPERIMENTS. oops! For my next trick I am going to try the 110SR module this weekend and see if i can get it to work.

Nick

[DrewPalmer04]

Congrats! I hope to figure out my issue by this weekend. You win the race for an Ethernet add-on lol

[rimai]

YAY!!!
Cool...
I was reading somewhere that their modules are 3.3VDC operational with 5VDC I/O tolerance.