At start of the 2021. NLnet decided to sponsor development of modular FPGA board ULX4M (Successor of EMARD ULX3S).
Goal of this project is to make board modular and add more peripherals. We are specially interested in adding fast SerDes peripherals that are now missing on ULX3S. Along with that future plans are also to create modules with multiple FPGA vendors so we can do more vendor neutral coding.
It all started with selecting connectors – you can check story here
We ware also interested in blender so Paula joined team – her story is here
After we selected connectors and decided to go with CM4 IO compatible pin-out we have started to disassemble ULX3S and create new smaller modular board that will fit into CM4 IO carrier boards.
Progress on SDRAM version
Checking if ULX4M-LS is TOFU compatible
We have bean contacted by Oratek regarding ULX4M compatibility with TOFU board.
After a week TOFU board arrived in Croatia, and it looks like ULX4M fits nicely on this carrier board!
Quick check if video and SDRAM are working
Few things before switching to DDR3
We needed to check does it make sense to leve SerDes pins on simple header and we have found really good example how we can use SerDes in RTL experiments
1.2V FPGA overvoltage hack needed but receiver works (transmitter will need some support from YosysHQ)
Luckaly there are few repositories that already did some CSI experiments
But when we started to work on CSI we have notticed that on ULX4M-LD v002 CSI pins are not connected to true differential pins.
We also needed I2C to send commands to the camera. Easiest way was to connect ESP32 directly to both cameras I2C
So here it is not perfect patch, but it should work!
Next thing was to get camera into stream mode, and for that we needed ESP32 and micropython
There is a lot of more info in closed issues but here is latest code that worked and started camera stream
After that we whare able to capture some strange images.
Idea was to start camera, wait some time, save one picture into internal FPGA memory, and then slowly transfer image back to PC.
That was proof that if we connect CSI to good pins we should be able to get picture.
We have tried to get reall cam picture, but for now it was to complicated.
Best we had when chess test is off was this:
We will continue to work on this on ULX4M-LD v001, and now we know what pins to use!
There is also small change in DSI signals(just few more resistors) - you can also check that in new schematics or in close issues.
From SDRAM to DDR3
Checking other opensource projects that are using DDR3 (Trelis board, OrangeCrab, Butterstick, DC-SCM)...
For DDR3 we will need aditional 1.35V power supply for DDR3
After placing and routing 1.35V looks like this
Removing SDRAM and placing DDR3 memory
DDR3 schematics is copyed from AntMicro DC-SCM (they have used Trelis board as a reference)
In that schematics there is aditional part for DDR3 termination - we have decided to skip those parts, and try to get DDR3 running without it.
After schematics, we need to place DDR3 on board and start routing.
Next thing was to select right pins, as some pins need to be true differential bidirectinal! Also two right banks needed to be set to 1.35V
DDR3 is also using 8 pins more then SDRAM
So we have faced new problem, there was not enough free pins on those banks - so we needed to remove some GPIOs and move them to the left bank, so we ended with rerouting almost complete board.
As for DDR3 routing we have first routed differential signals, and after them all single ended signals.
After Routing was done we needed to match all signal lengths.
DDR3 track length
For checking all lines length I have used action plugins
In kicad you first select Route-Tune Track Length, click on longest track and remember length.
After that use right click for options select – Length Tuning Settings – you can also press CTRL+L to set settings screen. On the Single Track Length Tuning screen set some length – it needs to be more or equal to the longest wire you want to tune, and that is why you need to write or remember the longest line length.
After setting parameters. Now you just click on any shorter line and just pull the cursor over it. You will get those fancy curved lines. You can use the keyboard: 1 2 3 4 for spacing and amplitude of line curves.
This is how routing looked when matched
After almost everything was ready I have asked if someone from community could help me and recheck if I did erything correctly with DDR3. And I got response from gatecat
Ups! I need to reroute those signals. I did check datasheets multiple times, but I have somehow missed this
This is frech DDR3 schematics with all changes needed
And just in case I do not make same mistake again I have checked if diamond will allow me to use pins I have selected
After one day I have left with just a few sgnals that was then bit harder to route
This is length match we need to get
After new routing I needed to do length match again, and here is how routing looks now
If you check bit closer, you will see that clk is bit off - but that is only because we have two lines that from DDR pins goes to capacitors, without those lines clk+- are matched....
EMARD had few more wishes that could make board more attractive.
- adding 8 LEDS instead of 4
- adding more buttons
- adding DIP SW
- adding USB connector on the board so it could be used even without baseboard (with USB bootloader)
- 2.54 mm header on SerDes TX and RX line
We have decided to fullfill all wishes!
Adding - and routing ethernet
After adding all those additional periferals we needed to check and route ethernet.
Chips are still not available, and we do not know when we could get some, but we have decided to go with 1Gb KSZ9031RNXCA that is also used on AntMicro DC-SCM board.
Some more tips and tricks
For some ECP5 chips 1.2V is needed instead of 1.1V so we have added one more resistor to schematics, and this jumper, so we can switch to 1.2V with just closing it!
JTAG is not used so often, so we have switched to just toplayer PADs
Adding test point for 1.35V on top of the board
Adding more test points that are connected to unused pins - and informative table with some line lengths
Buttons are really small so it may be better if they not in line as then it is easyer to press one buttons instead of two
Kicad 3D before ordering
This time we wanted to order boards from EU
So we have tried to use https://www.eurocircuits.com/
But after 5 tries we have give up on that one. They have lots of rules, and if fallowed this board would be really expensive 3 pcbs > 1000€
At some point we almost managed to pass with 230€ for 3 boards, but one rule did not pass manufacturer check, we have asked if they can just ignore the rules and wrote that we understand that it can make boards unusable. But they are not able to produce any boards that is not passing all checks they have.
After one week of eurocircuits try, we have tried with https://www.multi-circuit-boards.eu
Price for 5 board was 113.40€ + 9.90€ for each stencil (top, bottom)
Along with that order that will arrive in Intergalaktik we also wanted to check new OSHPark 6 layer service
So we have created special version for our long time supporter in US GoJimmyPi
We must nottice that OSHPark has really good price for 6 layer boards!
Remider on JLCPCB prices
5pcs. 4 Layer price €63.30 – ENIG – BLACK – 0.2/0.4mm
5pcs. 6 layer price €129.91 – ENIG – BLACK – 0.2/0.4mm
Stencil is €10.54 + it adds up €20 on shipping
Total JLCPCB costs are:
10 pcs. ULX4M-LS v002 €140.16
Stencil with frame €10.29
5 pcs. ULX4M HAT €6.96
Total 10 boards with stencil including Tax and import fees: €257.47
Intergalaktik has stencil printer for big stencils, but this time we will receive small stencils, so we needed to check if there is better way of applying paste the to tape stencil on table.
Radiona member had one soludion that could work with those small stencils, so we have decided to give that one a try
But first we needed to do a bit of adjustements, as this will be double side board we need to cut some
Stencil holder was done with stencilframerm and aditional cut was done in FreeCAD
After a week boards arrived
But whaaat! There is no Silk screen!
But at least board fits in printed stencil framer, and stencils are the right size!
Problem with silkscreen is one checkmark - Marking print
Default is set to: without :(
Almost at the same time boards from OSHPark arrived to GoJimmyPi and those are looking great!
And they have marking print by default :)
Soldering paste time
Doing bottom side of the board with stencilframer was not possible.
After multiple tries we have switched to old way, with just taping stencil on table and applying paste.
Problem with stencilframer was just hiroshe connector - it was impossible to get that connector right :(
Top side was easyer to get with stencilframer
Again - Interactive HTML BOM was big helper in assembly, and after some work here is what we have
Lets start some LEDs
CSI0 to GPDI video
Just a quick test with already writen core shows us that CSI will now work. As we now have GPDI pins free it we can show picture from camera on monitor or TV
As you can see there is lots of things needed to get normal camera image, but this shows us that HW is good to go!
What is next?
Checking rest of the HW
- USB - power is ok, but something is wrong - emard has notticed that I have switched some USB lines - will need to cut and solder to check if it will work then, and then fix in next revision
- CSI1 - something is not right - I can get clock, but not picture - maybe some pins on connector are not soldered right
- DSI0 and DSI1 - that will take some time as there is not really good example how to do it easily
- GPDI0 and GPDI1 are working
- JTAG - serial are ok
- we need to check is SD CARD still ok, along with SDCARD sharing with ESP32
- DDR3 - now that we have it on board we will nee to spend some time in getting it running
- PCIe - opens a new world
- All SerDes pins are now connected to CM4 connector but we will need custom carrier board to check if some good things could be done - if Ethernet is used then there is no possibility of using some SerDes pins.
- Ethernet - we still cannot find any chip, just to check if everything on HW is working
After checking and confirming all HW is good we need to bring all changes from LD version back to LS version of the board
So there is lots of work ahead!
By: Goran Mahovlić
This project is funded by NLnet https://nlnet.nl/project/ULX3M/