There are times in life when you just have to check and make certain that what you believe is happening - really is happening. I was having a bear of a time figuring out why a particular hardware design wasn't working, hours spent pouring over the hdl and rewriting the verilog (which was a good thing to make it more efficient) didn't help... The simulations came out perfect, but alas when connected to the real world the design just didn't work. (bigger image)
In this particular design we are using an open drain for twi/i2c communication across multiple chips. In this type of interface we have multiple devices connected to the same bus and the way they act is by pulling the bus to a ground state (interpreted as 0). To get the line off the ground state when no device is talking, we use a pullup resistor to Vcc.
Typically I like to use 10K Ohm pullup resistors (probably got away with them more times than I should have) because the higher the resistance, the less power is lost in an effort to keep the line high. The problem this time was that 10K was insufficient (as I found out). Then again I am usually using hardware designed by another (microcontrollers/memory/displays) rather than myself. In this case I am using a Xilinx FPGA. Eventually I got a simple scope (Rigol DS1052E) and in only a few minutes figured out the problem.
On the top of the figure you can see the signal we intend to send - it's a nice 50% duty cycle square wave. In the middle of the top row you can see what the actual voltages look like on the bus... not so square a wave here. There is a sharp drop from high to low on the front edge when the chip grounds the bus, but when the chip releases it, the bus gradually returns to high (a pretty curve, but not a good thing here). On the right of the top row you can see the error (this is high when the chip sees a difference on the line between what it intends to send and what it sees on the bus). The figures on the next row measure either the intent and observed bus at the same time, or the observed and the error at the same time.
The good thing is that it's easy to fix... by using a weaker resistor we can pull the line up faster (the slope of the line is dependent on the difference in voltage and the amount of current we can draw). It should be obvious that the slopes are much sharper and the errors between intended and observed signals are smaller here.
This isn't anything surprising really - but it's an embarrassing reminder to me that it's usually the simple solutions that I "knew/should have known" and not the hard/complex ones that cause me the most problems...