lördag 12 augusti 2017

FuseSoC 1.7

Lock up your wifes and daughters! FuseSoC 1.7 has been unleashed on the world. This unstoppable force will organize your HDL dependencies and provide abstractions to your EDA tools without giving you a chance to defend yourself.

Actually, there's not that much new on the surface of this release. Most of the work has been spent on internal refactoring in order to bring in two new major features for the next cycle. The first of these is a separation of the frontend - which handles reading core files, maintains the core database and does dependency resolution - and the backends - which launch the EDA tools. There are several reasons for doing this, but I hope to write more about this specifically in another post. The other major feature is the preparation for a new core description format, called CAPI2. This will be added early in the FuseSoC 1.8 cycle, so expect to read more about this in the future as well. If you are interested in taking an early peek, there's a CAPI2 branch of FuseSoC together with corresponding branch of fusesoc-cores which is used as a playground for now.

So, onto the actual changes.

Test coverage has now reached 72% of the code base. Unit testing is something I should had done from day one, as it has uncovered plenty of bugs and been a huge help when doing refactoring. So kids, get tested you too!

Failure is always an option, and should be handled with the same loving care as success. FuseSoC now exits with an error code when a build or simulation fails, making it easier for external tools to pick up failures. Also, failing scripts now print out the error code on failures to make it easier to analyze what went wrong. Speaking of things going wrong, the parsing of the core files have been made improved to warn for syntax errors instead of leaving the user with a Python stack trace. In general, there has also been many improvements to the logging, so that running with the --verbose option might actually be helpful when debugging strange errors.

There have been a number of improvements in the tool backends, mostly related to parameter passing. The Vivado backend had a bug that prevented passing multiple parameters to the backend. Quartus now supports passing verilog defines on the command-line. Parameters are properly escaped before being passed to the backend, which fixes string parameters for some backends. Other than that, ISIM now supports multiple toplevels, which is required for example when simulating Xilinx primitives that require glbl.v as a parallel toplevel. The Vivado flow now works on Windows after discovering that Vivado prefer forward regardless of what the OS uses as path separator.  The Icarus backend has been rewritten so that it's easier to rebuild the simulation model from an exported build tree.

In addition to fixes and new features, a few features have been removed. Mostly because they made no sense, were broken or turned out to be hard to maintain with little gain. The system-info command is removed, as all details are shown in core-info anyway. The submodule provider was likely broken for a long time without anyone complaining, and was a bad fit for the FuseSoC depedency model, so it has been removed too. There was also a semi-working feature of the verilator backend that aimed to convert files containing verilog `define statements to a correspondent C header file. As there might be users out there actually using this, I added an entry to the FuseSoC migration guide with information on how to replicate this functionality in newer versions of FuseSoC.

Other than that, there are some other bug fixes, like FuseSoC now supports putting IP-XACT files in subdirectories of the core tree. There is also a --version command-line option to show, surprise, the current version of FuseSoC.

That's more or less it. Make sure to upgrade and get prepared for the wild ride that will be FuseSoC 1.8

Peace out!

tisdag 21 mars 2017

FuseSoC 1.6.1

This is just a quick one. I was asked by fellow FOSSi Foundation founder and OpTiMSoC creator and lead architect Stefan Wallentowitz to do a point release. OpTiMSoC is currently shipping a forked version of FuseSoC, and while doing some spring cleaning they wanted to see if the upstream version could be used instead. As there has recently been some problems with the FuseSoC dependencies, originally caused by python's sorry excuse for a package manager called pip, this was also a good time to put out a new version. So here it is. Not much changed on the surface, but there has been some refactoring and preparation that is needed for the upcoming CAPI2 core format, which eventually will supersede the current format for .core files. If you have any thoughts about what the new format should look like, please check out the current CAPI2 draft, and yell if something needs to change. Other than that, the most noteworthy addition is that FuseSoC now comes with unit tests. Not sure I like them though. The first thing that happened is that they pointed out a bunch of bugs that I didn't know about. Oh well. I'll get those fixed eventually. That's all. Have fun with the new release!

lördag 7 januari 2017

The dream of HDL standard libraries

I was recently asked why RTL code is so unportable and why there aren't any standard components to use for common blocks like RAM and ROM. As this is something that I've been thinking about for a long time, and was part of the reason why I started working on FuseSoC, I started writing a long reply, but realized it was turning into a decent-sized blog post instead, so I decided to put it here instead. So here are my thoughts on the issue, why it is an issue and some ways to make it more manageble. My view on this is largely FPGA-centric, but much of it is applicable also for ASIC.

In the RTL world we have no standard libraries. Coming from a software world, this is something that very much expected to exist. There are plenty of more or less standardized software components and interfaces. A prime example would be the C standard library (libc), which is used by pretty much to some extent by all software written in C. It's portable, and the same functions can be used from the smallest microcontrollers to the beefiest server parks. Other programming languages have their own built-in libraries, but most of those are internally using libc deep down. So why don't we have the same thing for HDL code? Both VHDL and Verilog has been around long enough for this to materialize. I'm not sure why, but my guess is that part of the reasons is that the pool of RTL engineers are much smaller than their SW counterpart and that most RTL projects has been done in isolation at companies that hasn't seen the value of agreeing with other companies on a standard interface. Another part of the problem is the general ignorance that is often seen in the digital design field. One of the most commonly used phrases for HDL code is that "it's a hardware description language, not a programming language". Over the years, this makes me more and more annoyed, and I really wish that schools and companies started telling this to new engineers. No, it's not a programming language, in the sense that it gets translated to a circuit description (which is not true for simulations), rather than machine-language code. This, however doesn't mean that RTL engineers should ignore all the best practices that has been developed for software. Most of the general ideas from software can be applied directly to HDL code as well, some need interpretation and a few doesn't apply at all. Standard libraries is one of those useful ideas that can be applied directly to HDL code, so again, why don't we have them? Surely, someone else must have thought about this before.

Yes, there has been a lot of attempts to build standard libraries but it's hard to standardize something that is already implemented in a million ways, and from what I see, no one seems to be interested in using existing solutions. Part of this is due to the fact that there hasn't been an easy way to share code, and this is a big part of why I started with FuseSoC, so that it would be easier to reuse existing cores. I'd say that the constant reimplementation is actually worse than what it first looks like. Writing these things are not necessarily that difficult. The hard part is a) proving their correctness, which means both good test cases and testing against a multitude of target devices and b) usability, in form of documentation and easy integration into other projects. Most of the standard library contenders I have found falls short in both these areas, and just contain a code dump.

Just being able to reuse code more efficiently isn't a silver bullet. There are plenty of examples from the software world where similar functionality is being reimplemented over and over again. But it does help. Another issue is also the scope. What should go into this standard library? Really small things like registers and muxes? No, those are probably best handled by the HDLs themselves. How about FIFOs, RAMs, ROMs, SERDES, clock domain synchronisers? Yes, these are things that are common enough to be used in a lot of places while still being complex enough to benefit from the increased testing that comes with more users. How about even larger components like UARTs, SPI controllers and caches? While I personally would sacrifice my left pinky (this is a big thing for an Emacs user) to never see another reimplementation of a UART or SPI master, I think they contain too much configurability to be put this into a standard library. I am however a big fan of modularity, and I would be happy to see these components being built using elements from the still fictous standard library. It should also be said that there are some examples of standardized interface in the HDL world as well, mostly related to verification, perhaps since this is an area more closely tied to software development. UVM is an example of this for SystemVerilog, and OSVVM would perhaps be its VHDL counterpart.

So let's pretend that a large enough group of developers agree on a standard library interface, the next problem is the portability. Such a library would preferrably have both Verilog and VHDL implementations. The main reson for this isn't compatibility, as it is possible in most cases to API make the API language-neutral. This of course has the drawback that many of the useful features in each language can't be used as they have no direct translation in other languages. The main reason is instead that the state of mixed-language simulations are still not very good. Many commercial tools charge a lot of money for this feature (even though the situation has improved quite a bit recently), and the Open Source tools are still not good enough for mixed-language. Where does this leave all the fancy new HDLs such as Chisel or Migen then? In the great tradition of reinventing wheels, of course they have also created their own libraries for these things instead of reusing existing Verilog libraries. To be fair though, it's not as if there was any good libraries to build upon. At least none that could be considered standardized.

If we ignore this for a while and assume we have implemented the library in several languages, or perhaps only need one, the next issue is the target technology. Different FPGA and ASIC technologies use different primitives for things like memories, FIFOs etc. The good news is that many of these constructs can be described in regular HDL code, and the EDA tools will happily convert them to the correct target implementation. This should always be used as the first option if posssible. Unfortunately, I see so many cases of engineers generating a vendor-specific IP for the simplest things. This has several drawbacks. First of all, it's not portable. Definitely not among vendors, and in most cases, not even among devices of the same family. For some truly idiotic reason, it's also usually bound to a specific version of the EDA tool it was generated by, which means it becomes a mess of upgrades or downgrades whenever someone is trying to use it with another version of the tool. Depending on the format of the IP core, it can be a pain to use together with version control systems and it often requires manual changes and regeneration whenever the smallest parameter (such as FIFO depth) change. A classic example of trying to save some time on implementation which ends up adding a large cost instead on maintenance and reuse. Trust me. I've been through this too many times.

Not everything can be described easily, efficiently or even at all with vendor neutral HDL code. Examples of things that are closely tied to target technology can be clock generation, I/O cells or special macros. The good thing here is that if these features are needed, you likely already know which chip or technology you're targeting and can afford to lose portability. These things are not supposed to be in a standard library anyway. A good practice here is also to try to keep these things close to the top-level so that the core functionality can be more easily moved between different targets. Other blocks might be possible to implement in many different technologies, but can't be expressed as pure HDL, since the EDA tools can't map them in a satisfying way. In these cases, there need to be backend-specific implementations, possibly with a pure HDL fallback for some tools. This is a common approach, but it's not really standardized how to select the correct backend. Many RTL engineers uses VHDL/Verilog generate statements for this, IP-XACT uses the "view" mechanism to switch between different files and FuseSoC currently uses something similar to IP-XACT, with a more powerful mechanism being in the works, inspired by the idea behind Gentoo's use flags.

So to sum it up, I'd love to see more standard libraries, as I think it's a good idea. Unfortunately, it's hard to make this come true, both for technical and political reasons. This doesn't mean we shouldn't try, and as FOSSi Foundation now plays a role in aiming to foster collaboration and open standards we might have a better chance than before. This won't happen automatically however, and in the meantime there are a few ways to iteratively improve the situation. Follow these short guidelines and the world will become a much better place

1. Whenever you start writing new basic functionality, take a look around to see if there is something that can be reused. It might look like more work up-front, but further down the road it will mean less maintenance, better documentation  and hopefully fewer bugs for more people.

2. Use pure HDL whenever possible instead of relying on vendor-specific IP. Again, the benefits will come from less maintenance, and in this case also improved portability

3. If you decide to write your own code, document it, add testcases and publish it through LibreCores so that other people will find it. This improves the chances that your code will be reused and improved by other people.

4. Use FuseSoC! In most cases it's quite simple to put together a FuseSoC core description file to go together with you component. This makes it easier for other people to reuse your code if they are also using FuseSoC. It also makes it easier for you to reuse other code if that code is already packaged to be used with FuseSoC

Let's show those softies that we RTL engineers also can collaborate and build awesome things together!

lördag 24 december 2016

FuseSoC 1.6

Ok, so everything is done. I've done enough testing to make sure the new features work as expected and none of the old stuff is broken. The sources are just waiting to be tagged and uploaded to pypi. The blog post about the new release is written, and all I need is a catchy introduction. Oh, I know. It's christmas, so I should write something funny about FuseSoC being the best gift this year. Or wait, I take a christmas song and change some words so that it's about FuseSoC instead. Or maybe not. It's not really funny, and I suspect someone has been making similar jokes for at least 2016 years. I need to do something else. You know what? I can't hold off this release another day just because I couldn't come up with a catchy introduction. No one cares anyway. Let's just get this over with. FuseSoC 1.6 is released and it's really good. Keep on reading to learn about the new features.

New dependency manager


One of the major uses of FuseSoC is as a package manager. Unfortunately it has always been quite bad at this task. Perhaps the biggest issue has been that unlike most other package managers, FuseSoC only supported exact matches of dependencies. Together with the complete lack of namespaces in Verilog, and to some extent VHDL, this could make package conflicts a nightmare in some cases


An example: A common core in the FuseSoC standard core library is vlog_tb_utils. At some point I wanted to add some more features to vlog_tb_utils, and release it as vlog_tb_utils-1.0. If a core wanted to use the updated version of vlog_tb_utils, it would not work unless all other cores also updated their dependency lists to use this new version as well. That's because the dependency chain would pull in both vlog_tb_utils and vlog_tb_utils-1.0, which contained conflicting files. What a nightmare. What is changed now is that listing a dependency on a core will now pull in the latest version of the core, so all dependencies would now get vlog_tb_utils-1.0. There are of course ways to force an older version, by instead depending on <=name_of_core or =name_of_core, which can be handy when a core has changed in an incompatible way. As most of the available cores that has already been written don't have an explicit version, FuseSoC will create a version according to the core naming rules. The version format is heavily inspired by Gentoo's portage package manager.

Writing a package manager is complicated, and since I really don't enjoy doing difficult things, I decided to look around and see what was already available. Most of the modules I found for package managment were too simple to be of any use, but the simplesat solver from Enthought turned out to meet my requirements. Mind you, it's not a perfect solution, as it is built for another use-case, so I had had to shoe-horn the code to fit into FuseSoC, but it seems to work fine, and has been running on the master branch of the git repo since this summer. The code was actually mostly finished before the FuseSoC 1.5 release, but I wanted to put out a release before adding any potentially backwards-incompatible changes. It should however be completely backwards-compatible, except for a few corner cases, where I could make changes to the core files to handle both the old and new systems.

The work on the new dependency manager also brought with it some other related changes. All cores are now also fully identified by a VLNV identifier. VLNV comes from the IP-XACT standard and means Vendor Library Name Version. All new cores are recommended to use the VLNV naming scheme, but old cores will be mapped into a VLNV name. The documentation goes through the core naming rules in more detail.

As mentioned above, the changes to FuseSoC are backwards-compatible, but older versions of FuseSoC will not be able to handle the new dependency and VLNV syntax. For this, and other reasons, I decided it was time to freeze development on the standard core library called orpsoc-cores. Instead, a new library is created here , which will supersede the old library. As I did not have time to finish the migration, and also because I'm not sure I want to bring all old cores to the new library, these two standard libraries will both be used for some time ahead. If you're doing a fresh install of FuseSoC and run fusesoc init, both libraries will be installed and added to fusesoc.conf. For existing installations, I recommend cloning the fusesoc-cores repo and add it after orpsoc-cores in your fusesoc.conf. By putting the fusesoc-cores entry after orpsoc-cores, it guarantees that all cores which exist in both libraries will use the one from fusesoc-cores. If you're unsure which one was picked up, just run fusesoc core-info on a particular core and check the Core root

Another change caused by the new dependencies is that Python 3.2 and 3.3 are no longer supported. Python 2.7, 3.4 and 3.5 will still work just fine.

Documentation has always been one of the weakest points of FuseSoC. About a month ago, I made a promise to not add any new features until I had written documentation. This of course, I did not manage to completely fulfil, but at least there is a much improved documentation available. The new documentation consists of a tutorial on how to get started with writing core files, how core names are parsed, how to deal with core libraries, a quick-start guide for running simulations, specific information for some of the backends and a migration guide for updating old core files to the current best practices.

There is still a lot to be written, but at least it's moving in the right direction. In total, there is a 148% increase in lines of documentation. Let's see if I can match that for the next release.

I also noticed that renaming the asciidoc sources to .adoc allowed github to understand that this is asciidoc, and renders it automatically, which has the great effect that the online documentation now looks good in a browser without having to convert it to html first.

New backends


This version has support for two new EDA tools

Xilinx Vivado replaced ISE for some time now, and this release finally adds support for the new tool. The code was written by fellow FOSSi Foundation director Stefan Wallentowitz, and was originally planned for FuseSoC 1.5, but because of some major refactoring, I decided to hold it off until this release.

On the simulation side, there is now also support for Aldec Riviera Pro, which is a commercial simulator, quite similar in scope to ModelSim, that seems to be gaining popularity.

Death to .system files


When I started out working with FuseSoC I had an idea of a clear separation between running simulations on cores and creating FPGA images out of systems. This led to the .core and .system files. Over the years, the role of the .system file has been less clear. It had some overlap with the .core files, was not properly documented, added some extra code paths and required an extra file. So from now on, the .system file is made optional, and it is recommended to move the settings from there into the .core files instead as stated in the migration guide

Internals


There are also many smaller but oh so important changes

FuseSoC will now pick up additional core libraries from the FUSESOC_CORES environment variable, and all core library locations can be listed with fusesoc list-paths

Both simulation and build backends have gained a --setup flag that will only export the needed files and setup an initial project file without running the EDA tools. This is very useful both for running the tools in GUI mode, or to export a build tree that can be released as a stand-alone archive for people who don't use FuseSoC.

It is now also possible to set verilog `define options on the command-line through the parameter interface. In the core file, it would look something like this

[parameter size]
datatype = int
description = Sets the size of something
paramtype = vlogdefine

and would be called with fusesoc sim some_core --size=10

I've had the opportunity to run some testing in a Windows environment which revealed a few flaws that has now been fixed. These were mostly related to the fact that even though Windows uses \ as the path separator (insane!), most EDA tools still required using / (sane!) even on Windows. Both the ModelSim and Quartus backends have been updated to reflect this, but there is a good chance the other tools need the same treatment. To make it easier to set up on windows platforms, there is also a change to use git-apply for patching cores instead of using the patch command. As FuseSoC already depends on git for setting up some things, it removes the number of external Linux-specific dependencies.

Speaking of ModelSim, there has been some improvements to make it easier to run in GUI mode and batch mode if FuseSoC is run with --setup or --build-only. FuseSoC now also supports multiple top-level modules, which is something that is often required for running with the Xilinx simulation libraries


There are probably more things that I forgot about, but they are likely not that important. So for now, please upgrade to FuseSoC 1.6 and try out the new stuff. I'm already hacking on some new things for the next version.

Have fun!

onsdag 2 november 2016

IP-XACT: The good, the bad and the outright madness

I've found myself in a strange position recently defending IP-XACT on a number of occasions. Most recently there were some heated discussions with lots of hand gestures at orconf. The heat and hand gestures however might have been due to the fact that we were in Italy. Still, there were discussions.

So let me get this straight once and for all by quoting one of the great thinkers of the 21st century.
IP-XACT is in most parts horrible, but it's the best chance we have for a vendor-neutral standard.

O. Kindgren, 2016

So what is this mostly horrible but still kind of useful standard? It's an IEEE standard developed by Accellera aimed to create an interchange format for IP cores. You can find the standard here, but the Wikipedia article does a better job at capturing the purpose and extents of the standard in a few sentences.

IP-XACT is an XML format that defines and describes electronic components and their designs. IP-XACT was created by the SPIRIT Consortium as a standard to enable automated configuration and integration through tools.

The goals of the standard are

  • to ensure delivery of compatible component descriptions from multiple component vendors,
  • to enable exchanging complex component libraries between electronic design automation (EDA) tools for SoC design (design environments),
  • to describe configurable components using metadata, and
  • to enable the provision of EDA vendor-neutral scripts for component creation and configuration (generators, configurators).

These are all noble goals, so why the criticism? First of all, it's an XML-based standard. This alone will turn away many people, but frankly, don't tell me that json is any better. For some reason though, json has managed to attract a lot less negativity while still being affected by some severe shortcomings and compatibility problems. My personal pet peeve is that there isn't any way to add comments to a JSON file, which is very annoying when you want to temporarily remove a part of the file, or explain something. For those of you who don't know json that well, it looks a bit like LISP code with wrinkles.

With that out of the way, another problem is tool compatibility. As with most standards in the FPGA world, the two giants, Xilinx and Intel (still haven't gotten used to saying Intel instead of Altera) does a superb job of never backing the same standard, to ensure an absolute minimum of interoperability. Prior examples being Xilinx endorsing the FMC standard for daughter boards with Altera going for HSMC. Xilinx used to have their UCF constraint format which didn't look like any other constraint format, but have now moved to XDC, which is almost but not quite compatible with the more common SDC format. In the case of IP-XACT, Xilinx supports it, but not Altera

Actually, supports is a bit too strong a word in this case, as Vivado only works with up to the 2009 standard, not IP-XACT 2014. ISE of course doesn't handle IP-XACT at all, but I guess that's ok since it has been obsoleted since some years. Actually, when I say that Vivado supports up to the 2009 standard I'm still being a bit too kind, and this brings me to the first issue with IP-XACT

Vendor extension

Vendor extensions means that tools are free to define their own non-standard XML tags in the IP-XACT files. This allows them to implement additions where they feel the standard is lacking. This can be a good thing, but the drawback is that it opens up for a lot of non-standard additions, where every vendor implements the same functionality in different ways. I've seen examples where most of the file is just encapsulated in vendorExtensions. Incidentally, this is a bit similar to what happened to the other vendor-neutral EDA standard, EDIF, where no two files from different vendors were really compatible with each other. Wikipedia has a fascinating article about EDIF, which should serve as a cautious reminder to the future of IP-XACT

Tool support

What about IP-XACT support in other tools then? Unfortunately most commercial IP-XACT tools are out of reach for someone not deeply entrenched in ASIC development, so my other examples will be open source tools. The likely most used open source tool for working with IP-XACT is Kactus2. Kactus2 is a graphical application where you can fill in the various fields of an IP-XACT component file in a more or less user-friendly manner. It can parse Verilog/VHDL code to help you import existing HDL code into an IP-XACT model. Once you have built up a library of files, you can hook them up together in a block diagram view and export the top-levels as verilog or VHDL together with documentation, C header files for registers and a few other things. Kactus2 up to version 2.8 only handled the older IP-XACT 1.5 version, while Kactus2 3.0 (yes, it's Kactus2 3.0, not Kactus3. I asked the developers) and onwards only handle IP-XACT 2014. The attentive reader will notice that this means that Kactus2 and Vivado are no longer able to work with the same files.

For my own humble uses of IP-XACT, I decided to write a small python library called ipyxact to easier interface the parts of IP-XACT that I care about. This means it only support a limited subset of the standard. On the upside, it reads and writes 1.5, 1.6, 2009 and 2014 files, and there are some ideas how to make it cover the full standard. So, as you can see, there are still tools to be written for IP-XACT. For the IP-XACT files themselves Accellera provides XSL scripts which can be used to automatically upgrade files to a newer version. (Hey JSON, you got something like XSLT? No? Didn't think so either)

What about the good parts then? There must be something good to say about IP-XACT too. Fortunately, there are some good parts, and these are enough for me to endorse it, even with its numerous other faults.

VLNV tags

Each named object, such as a component or bus in IP-XACT have an identifier called VLNV. VLNV stands for Vendor, Library, Name, Version and is used to uniquely describe an object. These are often written as a colon-separated string, such as librecores.org:peripherals:uart16550:1.5 Even though it sometimes can be a bit awkward to come up with a good naming scheme for Vendor and Library, it will work out fine for most use cases, and the four fields are enough to avoid any namespace conflicts. Well done IP-XACT!

Block diagrams

One part of the IP-XACT standard allows you to define the interfaces of your IP cores. This can be single wires, but more common is to group these together into a bus, such as Wishbone or SPI, which can be connected between components. There are several graphical editors to help with this too. Kactus2 is the main example, and is also Open Source. Vivado comes with a built-in block diagram tool which is based on IP-XACT too. I love this part of the standard. I always draw my designs on paper before getting to work, so why not be able to use this for both documentation and implementation? I know that many of the new HDL such as Chisel and Migen was born partly out of frustration of connecting components in Verilog and VHDL. These languages makes it a bit easier, but it's still programming code, and in my opinion this is a graphical problem best solved with a graphical tool. Do your business logic in your language of choice, but leave the interconnections to a drag-and-drop application. Kactus2 still has some ways to go to make this really comfortable, but it works fine for many cases and I have done a few Proof of Concept designs to get better acquainted with the process. The drawback here is that it's not that flexible. Even though interfaces can be parametized with regards to widths and types, I haven't found a way to create components with a variable number of buses, which is handy for things like bus interconnects where you want to quickly hook up or remove master and slaves. My solution for that is to use my trusty ipyxact library to generate application-specific wrappers for these components. It works, but requires an extra step. There is unfortunately another problem with the way bus interfaces work in IP-XACT. Each bus type is identified by a VLNV tag. This means that a one_vendor:buslibrary:busname:version is not compatible to other_vendor:buslibrary:busname:version. And you know what? IP vendors are happy to put their own names on things, but not their competitors, so this will happen a lot.

In all fairness, this problem comes down to management of the shared resources and Accellera are providing bus definitions for a few common buses, like SPI and the various MII variants to connect ethernet Phys and MACs, but it would be great if other owners of buses would do the same as well. For LibreCores we are planning to host open standards such as Wishbone and we're happy to help if other want to host their standard bus defintions here as well

 File sets

File sets lists the files of an IP. Each file contain information such as file type, if it's an include file, which library it belongs to and other metadata. The filesets themselves have additional properties such as describing its purpose. This can for example be used to expose different filesets for simulation or synthesis. The filesets are actually the only part of IP-XACT that is currently supported in FuseSoC. Instead of listing the files directly in the FuseSoC .core files, a .core file can point to an IP-XACT component file and FuseSoC will automatically pick up the files from there. And even if you prefer to use only .core files, the fileset sections in the .core files are modeled after the structure in IP-XACT. Sounds good? Mostly, yes, but there are some drawbacks. First of all, the files can contain a lot more metadata than what is listed above, but I will return to that later. Instead, I will focus on the file types. The IP-XACT standard decided to enumerate all valid file types in the standard. Here's the complete list for the latest (2014) revision.

unknown, cSource, cppSource, asmSource, vhdlSource, vhdlSource-87, vhdlSource-93, verilogSource, verilogSource-95, verilogSource-2001, swObject, swObjectLibrary, vhdlBinaryLibrary, verilogBinaryLibrary, unelaboratedHdl, executableHdl, systemVerilogSource, systemVerilogSource-3.0, systemVerilogSource-3.1, systemCSource, systemCSource-2.0, systemCSource-2.0.1, systemCSource-2.1, systemCSource-2.2, veraSource, eSource, perlSource, tclSource, OVASource, SVASource, pslSource, systemVerilogSource-3.1a, SDC, vhdlAmsSource, verilogAmsSource, systemCAmsSource, libertySource, user

Quite some list, right? Both SystemVerilog 3.0 and SystemVerilog 3.1 is there. There are also something called executableHdl which I have no idea what that is. Looks like everyting is covered. Well, what about verilog 2005? Not a major language revision, but they added $clog2 in that version which is quite handy. Or VHDL 2008? That one is really big. Turns out that they forgot about those. There are also a gazillion new HDL languages such as Chisel, MyHDL, parC as well as all the vendor-specific IP formats, which have no entries in this list. For FuseSoC, I have added a few new file types (currently QIP, verilogSource-2005, vhdlSource-2008, xci and xdc) which have been required, which already makes it a superset of IP-XACT. Enumerating types like this is just a bad idea, and we will see even better examples of this later on in this article

 Register maps

Register maps is one of those areas where you want to make sure that the documentation matches the implementation. For this reason it makes sense to have a single source format from which you can create nice looking tables, C header files, slave interfaces for various CPU buses and perhaps test vectors. Most projects I have been working on have had different strategies for this. Many just do an implementation for a single bus interface and use a separate document where the designer has to do the tedious and error-prone work of writing down the same information and remember to tell the software developers when and how to update their C header files. This is of course very often not kept in sync. One magnificent example is a company, which shall be left unnamed, which stored all their registers in a spreadsheet file which was put under version control as a binary file and parsed this file with some magic scripts to extract the information. Nothing has ever gone wrong when parsing spreadsheets, right? And comparing two versions of a spreadsheet is always a joy.

IP-XACT, while being a XML-based format which some smarty-pants will always claim is actually a binary format, makes it a lot easier to keep the different target formats in sync. The register maps in IP-XACT capture enough information to describe most common access types, such as read-only, clear-on-write, single-shot and different bits slices of a register can be fully independent from each other. As usual with IP-XACT it's also possible to set some less common properties, such as marking a register as readWriteOnce. Might be useful if you're implementing a CD-ROM perhaps, I don't know.

Let's discuss the drawbacks then. Well, it turns out that the IP-XACT standard committee themselves didn't think register maps were good enough, so they came up with yet another format called SystemRDL, which they recommend using to *drum roll* generate IP-XACT register maps! So despite having added all these really complex features to express all details about registers they recommend moving to another, even more expressive format, to create the register maps. Maybe they should consider, perhaps, dropping register maps all together from the standard if that wasn't good enough? It's not like IP-XACT would be too small without the register parts. Instead, it seems like they are adding more features to registers for every standard revision. And this thing about adding features to the standard brings me to the really scary parts. Let me present:

IP-XACT as a build system

Every file listed in a fileset has some metadata, as I mentioned above. One of these metadatums (not at all sure that's the correct word for a single piece of metadata) is the buildCommand. It is a whole XML tree structure used to describe how to build said file. It comes with a ton of complex options used to describe which flags to use for building and then linking the file. Flags can be taken from a default build command, overridden by the build command specified for the file or taken from its sibling files, all according to some vaguely defined rules. Having a single buildcommand is of course impossible to combine with tool-independence since there is no standard command to deal with HDL files. Each tool has its own syntax and structures. Embedding this in the IP-XACT files is just wrong! Ok, let's assume that this isn't for HDL files then. Since building and linking is explictly mentioned in the tag names, we can guess that this is aimed at C code. Problem is, this can work totally different for other languages. Even for C files, this is a problem, since it's not really possible to switch between gcc, llvm or other compilers. There are also no way to do conditional compilations based on which flags are being set at a higher level. Please, please, please IP-XACT. Leave this task to experienced software developers. There is already an endless list of build systems for software (autotools, cmake, meson, scons, waf...) with decades of work to get them to the state where they are. Many of them still have some really bad limitations and the last thing the world needs is another half-assed build system designed by RTL designers who's only exposure to software structure was taking a class in Java back when Spice Girls topped the charts. This is unfortunately all too symptomatic of the EDA standards, and no amount of craziness in the IP-XACT standard manages to come close to the proposal to add inline Perl in SystemVerilog. Sometimes I think we're just doomed.

Speaking of software, this leads me to another favorite part of IP-XACT

IP-XACT to describe software API

Digging deep into the dark corners of IP-XACT, you will find that each fileset can be matched to a "function". The documentation of this particular tag says "Generator information if this file set describes a function. For example, this file set may describe diagnostics for which the DE can generate a diagnostics driver". Ok, not all that clear what this means, but let's move on and see what's beneath this function tag. There is a tag called entryPoint, which apparently is the Optional name for the function. There's also a fileRef, which is used to describe A reference to the file that contains the entry point function. Likely, those two are related somehow. There are a couple other tags as well, but the real gem is returnType, with the helpful documentation saying Function return type. Possible values are void and int. Yes, as we all know, int and void are the only two types a function will ever return. Let me illustrate this by a short conversation

- Hi Mr ipxact_square_root_function!
- Hi!
- Do you think you could give me the square root of three
- Certainly! It's my pleasure. The answer is.... let me think... one!
- One?
- Yes, one.
- Are you.. I mean, shouldn't that be more like... is that really...?
- Look, you should be happy that I gave you something. Or would you prefer I give you void instead?
- No no no! One is fine, one is fine. My bad
- I thought so
- (grumble grumble, I'm not asking that one again)
- What did you say?
- Oh! No, nothing. One is fine. Moving along

So my point here is that large parts of this standard manages to be extremely complex, while at the same time having strange limitations that makes them mostly useless. Couldn't we just fix these parts of the standard then? Well, this leads me to


IP-XACT, the free standard





To take part of the decision making progress, all we need to do is becoming members of Accellera, which can be done by registering on their home page. Oh, and you need to pay $25000 too. As much as I'd love to take part in steering this standard away from it's very uncertain path, I have a hard time imagining myself going to my boss and say "Hey, I need $25000! Why? Someone is WRONG on the internet. I need to fix this". Instead we have participants from the usual suspects in the EDA industry, and ideas from academia and the general software industry are nowhere to be found



The verdict

Having worked as a contractor on FPGA/ASIC designs for many years as well as digging into dozens of different Open Source Silicon projects, often with the purpose of packaging them for FuseSoC, I have come across many many, ways to do the exactly same thing. I'm dead tired of having to learn yet another register map generator or IP core description format only to find that I need to write another set of tools to automate the process. It's incredibly time-consuming, and this is why I cheer for IP-XACT, despite its numerous faults. It's a standard way of dsecribing IP cores. Given some time and effort, I think that there will grow a healthy ecosystem around the standard, and I think in fact that it's absolutely required to have everything from full-blown environments to small utilities for dealing with IP-XACT. Otherwise, no one will see the point of supporting this multiheaded beast of XML.

I also hope that the EDA industry will come to its senses and see that much innovation and knowledge never reaches the standards as long as the organizations charge these sums of money to participate. Also, if the standard continues to branch out with more of these half-baked ideas, it will die under its own weight. If that happens I hope the good ideas will be forked off to a new standard, hopefully driven by a more inclusive organization. But I hope that won't happen at all, because it would likely result in ten similar standards fighting for attention, and as I mentioned in the outset, this is the best chance we have for a vendor-neutral standard. Let's go with the flow and see where it takes us. We can always jump ship later.

måndag 8 augusti 2016

FuseSoC 1.5

Finally! FuseSoC 1.5 is now released. I was just about to release it a few weeks ago when I discovered two quite serious bugs. Then I was just about to release it when I had written a lengthy blog post describing some of the new backends. At that point I realized that it would be better to write those bits as part of the documentation. Yes, you read it right. Documentation. This release has 45% more documentation than previous releases. A massive increase from 245 lines to 368 lines. Oh well. One day it will all be properly documented...I hope. At least the new backends are documented and I will try to give the old ones the same treatment. Promise!

Now for the good stuff...

 New backends


This version adds support for one new simulator, one new FPGA flow and three EDA vendor IP package formats.

The new FPGA flow is for the awesome project IceStorm, which provides a complete open source flow for building FPGA images for Lattice iCE40 devices. The tools for this consists of yosys to handle synthesis, arachne-pnr for place & route and icepack for creating the binary file. Both yosys and arachne-pnr can handle other vendor's devices as well, and I hope to use these tools in other flows as well in the future.

There is support for an open source VHDL simulator called GHDL, which is most likely the open source EDA tool with the best VHDL support.

The new IP package formats being handled are Altera QIP files, Xilinx Coregen files (for ISE) and Xilinx xci files (for Vivado). The last two deserves a few extra words since they use the provider mechanism as a plugin system in a clever way. Providers, in the FuseSoC world, are modules that are normally responsible for downloading a core and put it into the cache. There are providers already for getting cores from git and svn repos as well as from simple URLs. The providers are each responsible for their own options but typically look something like this

[provider]
name = github
user = olofk
repo = wb_bfm
version = v1.0


One FuseSoC user and contributor however realized that the provider doesn't really have to download anything, and can instead be used to do transformations on source code instead. He implemented this to generate HDL code from vendor IP core descriptions and added the new coregen and logicore providers. They each look like this

[provider]
name = coregen
script_file = <coregen ip description>.xco
project_file = <coregen project file>.cgp


[provider]
name = logicore
script_file = <generator script>.tcl
project_file = <vivado ip description>.xci


When a core with one of these providers is requested, the Xilinx IP generation tools for ISE or Vivado are invoked to generate the HDL code, which is then placed in the cache. For the coregen-based cores, the generated HDL files are listed in the .core file as would have been done for any core.

For the Vivado-based logicore IP however there is an extra trick. As Vivado generates an IP-XACT component description for its generated IP cores, we can simply point to that one, and automatically let FuseSoC get all the HDL files for us automatically.

I have been planning for several years to do some kind of plugin mechanism to run custom preprocessing commands on source files, and using the provider mechanism for this solves the problem in most cases. Examples of future providers could be to turn MyHDL, Migen or Chisel code into verilog, or build a verilog top-level from an IP-XACT design file. As this was not originally my idea, I was naturally extremely sceptic of the whole idea. It does have some limitations, but for most cases it gets the job done and is a clever solution for the problem.

Continious integration


Another great addition brought in by a new FuseSoC user is the continious integration testing with Travis and AppVeyor. Every time something is commited to the FuseSoC repository, Travis and Appveyor are triggered to run some basic tests. As an interesting note, it took less than a week after adding the CI support until this found a bug caused by a minor difference between python 2 and python 3. That bug would probably had gone unnoticed for a long time if not for this, so it's worth its money already. Especially so, since both services are free to use.

Windows compatibility


The same user who brought in the CI testing also done some changes to the build system to increase the windows compatibility. It still doesn't run properly on Windows, but it can now at least install, list the cores properly and run a few simulations

Improved parameter handling


There had been several improvements to the handling of command-line parameters for compile-time or run-time configurations. It's now possible to specify a default value for parameters. This can be useful to override default settings in the upstream code, for example to specify a different path to a memory initizalization file than what the upstream project uses. There is also some visual improvements to the output of fusesoc {sim,build} <core> --help to make it clearer what kind of parameter is being changed. Several bugs has been fixed in this area as well, and it should now be possible to use top-level parameters with all tools.

The old [plusarg] sections that were deprecated in favor of the [parameter *] sections are no longer parsed as I have decided that users have had enough time to migrate to the parameter sections. The warning still remains to remind everyone that it's time to move on and the article about FuseSoC 1.4 has more info on what the [parameter] sections should look like.

Improved ModelSim backend


The Modelsim backend has been rewritten, both to support loading user-specified TCL files and to make it easier to rebuild a design from the exported sources. And believe it or not, there is even documentation for the modelsim backend now!

Build system changes


Finally, I also decided to remove the old autotools scripts for installation. For some time it has been possible to install FuseSoC with either python's setuptools (e.g. pip install fusesoc) or autotools (e.g. ./configure && make && make install). With the improvements in the setuptools based system, it makes little sense to keep autotools. I don't want to keep two tools updated, so from now on, FuseSoC will only use setuptools for installations. Farewell autotools. It's not you, it's me

And...


...as always, tons of bugs were killed during the making of this release. A few notable ones are that VHDL and SystemVerilog are now supported properly in ISIM, IP-XACT 2009 and 2014 files are supported, RTL library affiliation is read from IP-XACT logicalName tags and FuseSoC doesn't crash anymore if it encounters an unknown simulator in the simulators list

I'm really excited to start working on FuseSoC 1.6 now. There are already a bunch of new things ready to be commited, that I have been holding off until FuseSoC 1.5 was released. Make sure to check the git repo for daily progress if you want to get all the new things before the next version is released.

Thanks for listening!

torsdag 2 juni 2016

FuseSoC and your custom workflow

Last week a colleague made a horrible mistake. He casually asked me to tell him a bit about FuseSoC. This made me very happy and after an hour he probably regretted asking (sorry Johan).

Anyway, it turned out that he wanted to know if FuseSoC was the right tool for him. As most other RTL developers, he had his own set of custom scripts to launch simulations - in this case it was Makefiles. After discussing a bit back and forth we both realized that switching to a complete FuseSoC-based workflow wouldn't really be all that great for him, as he would lose some of the tight tool-integration with his existing scripts. Talking a bit more however revealed that he had no great solution for bringing in all the source files that his makefiles should digest, and as I have written about before, FuseSoC is not only meant as a end-to-end solution for running simulations or making FPGA images. It's also meant to be used as a library so that it can be hooked up to other tools. And what my colleague needed in this case was mainly a list of all RTL source files, so I said that I could make such a tool based on FuseSoC in fifteen minutes. It turned that I was wrong. Starting out with the script I did for VUnit integration, it took less than ten minutes to get it working. Without further ado, I therefore present the FuseSoC file dumper script as an example on how to use just the parts of FuseSoC that brings in dependencies, sorts them in order and lists all files, as long as your cores use the FuseSoC .core format to describe them. Use it, adapt it and share it.


#
# FuseSoC source file dumper. fusesoc_file_dump_demo
#
# Copyright (C) 2016  Olof Kindgren <olof.kindgren@gmail.com>
#
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

from collections import OrderedDict
import os.path
import sys

from fusesoc.config import Config
from fusesoc.coremanager import CoreManager, DependencyError

if len(sys.argv) != 2:
    print("Usage: {} <top level core>".format(sys.argv[0]))
    exit(1)
top_core = sys.argv[1]

#Create singleton instances for core manager and configuration handler
#Configuration manager is not needed in this example
cm = CoreManager()
config = Config()

#Add core libraries that were picked up from fusesoc.conf by the config handler
try:
    cm.add_cores_root(config.cores_root)
except (RuntimeError, IOError) as e:
    pr_warn("Failed to register cores root '{}'".format(str(e)))

#Get the sorted list of dependencies starting from the top-level core
try:
    cores = cm.get_depends(top_core)
except DependencyError as e:
    print("'{}' or any of its dependencies requires '{}', but this core was not found".format(top_core, e.value))
    exit(1)

#Iterate over cores, filesets and files and add all relevant sources files to a list
incdirs = set()
libraries = []
src_files = []

#'usage' is a list of tags to look for in the filesets. Only look at filesets where any of these tags are present
usage = ['sim']
for core_name in cores:
    core = cm.get_core(core_name)
    core.setup()
    basepath = core.files_root
    for fs in core.file_sets:
        if (set(fs.usage) & set(usage)) and ((core_name == top_core) or not fs.private):
            for file in fs.file:
                if file.is_include_file:
                    incdirs.add(os.path.join(basepath, os.path.dirname(file.name)))
                else:
                    libraries.append(file.logical_name)
                    src_files.append("{},{},{}".format(os.path.join(basepath, file.name),
                                                    file.logical_name,
                                                    file.file_type))

print("==Include directories==")
print('\n'.join(incdirs))
print("==Libraries==")
print('\n'.join(list(OrderedDict.fromkeys(libraries))))
print("==Source files==")
print('\n'.join(src_files))

Granted that you have a fusesoc.conf file in your current directory or ~/.config/fusesoc and the FuseSoC standard core library installed, both of which you get if you install FuseSoC, you can now run python fusesoc_file_dump_demo.py de0_nano to get a list of all source files, include directories and libraries needed to build the de0_nano system.