Hello,
My company is devloping a signal processing board, and I have been
tasked with writing all of the software for it. Here are the details.
We are developing a PMC board that will be usable with a PCI or VMEbus
carrier. The board is built around a Texas Instruments TMS320C6416
DSP, and its expected use is in conjunction with a single-board
computer running either Windows XP/2000 or Linux. It has 16 MB of
onboard SDRAM that is accessed through an EDMA controller. (This
memory block starts at address 0x80000000 on the board's local bus.)
The host processor will control our board by memory mapping this block
of RAM. Conceptually, it sound pretty easy, but I've hit some snags
while trying to figure it out, namely the setup of the PCI
configuration registers.
To make the aforementioned SDRAM block visible to the host processor
on the PCI bus, I would configure the first base address register as
follows: Set the address space bit to 0, indicating that this is
memory space. Set the memory type bits to 00, indicating a normal
32-bit address, set the prefetchable bit to whatever, and use the
remaining bits to indicate the size of the memory block - in this
case, 0x01000000. When the system boots, the BIOS will do its
plug-and-play thing, and allocate 16 MB of space for my card on the
bus, writing the address into the configuration register. For example,
suppose my card was placed at address 0xA1000000. A request on the bus
to read the data at address 0xA1ABC000 would then be sent to my board,
where it would be translated to a read of address 0x00ABC000 on the
local bus. However, I want that to be 0x80ABC000. So, how do I
indicate that the memory block starts at 0x80000000 instead of
0x00000000? Or am I completely misunderstanding the PCI system?
Also, what about VMEbus? There's a lot less information on the web
about this bus than PCI. I've been able to find out that the card's
bus address is hardcoded, unlike the dynamic PnP system in PCI, but
I'm unsure if I have the same problem that I stated above. Could you
describe the VME addressing scheme for me?
Finally, I am trying to determine whether or not we need to write
device drivers for the board. I've never done this, and from what I've
gathered, it's not a trivial task, even for a simple memmap driver.
I've purchased Walter Oney's "Programming the Windows Driver Model,
2nd Ed" and Rubini & Corbet's "Linux Device Drivers, 2nd Ed", so I
have an idea of what I would need to do, but I've heard of people
accessing hardware without using a custom driver. I've found
references to using the /dev/mem filesystem to access devices in
Linux, but it's apparently not reccommended for serious use. And I
don't know of anything comparable in Windows. What's my best course of
action?
Note: I can't actually test anything yet, because we won't have any
usable hardware for a few weeks. I'm just trying to get as much done
as I can beforehand. |
Request for Question Clarification by
maniac-ga
on
10 Jun 2004 14:08 PDT
Hello Ade3ada,
Hmm. A very nice set of questions, but I am not quite sure any of the
researchers can answer all of it. For example, I could:
- make some suggestions related to the address mappings on the board
[though you should describe which (e.g., vendor / part number) part
you are using for the PCI interface / EDMA controller]
- provide some background on VME addressing methods, having dealt
with VME to PMC interface cards
- make some suggestions related to Linux device drivers,having worked
with a couple
but cannot provide any guidance on the Windows part of the situation.
I could proceed on answering those parts if you are interested. If
not, and you don't get a firm answer in a day or two - I suggest
breaking this up into individual questions (with an appropriate price
for each).
--Maniac
|
Clarification of Question by
ade3ada-ga
on
10 Jun 2004 15:21 PDT
Thanks for responding, Maniac.
Since it sounds like you are familiar with everything but Windows
drivers, I would be very happy if you would just answer what you can.
Presently, we're looking at using the Sabtech Pelican VMEBus
(PC-SP012-00) and PCI (PC-SP001-00) PMC carrier cards
(www.sabtech.com). The PCI interface and EDMA controller are both
onboard the TI DSP chip. TI's documentation can be found here:
http://focus.ti.com/docs/prod/folders/print/tms320c6416.html I've read
the relevant PDFs several times now, and I didn't see any information
about the PCI mapping. I didn't actually expect to, though, as this
seems to be the sort of thing that would be covered in the PCI
specification. Either a memory block that is exposed by a PCI board is
assumed to start at address 0, or there is a standard way to indicate
the block's starting address on the local bus. Please correct me if I
am wrong.
|
Request for Question Clarification by
maniac-ga
on
12 Jun 2004 08:04 PDT
Hello Ade3ada,
OK. I checked the web sites you noted - for Sabtech, I had to send
them a message and will need one or more replies to make a firm answer
but can comment on what I expect in the mean time.
Let me use some ASCII art to illustrate the situation as I see it -
please correct if needed:
PC CPU Memory
+----+---+
|
Host Bridge
|
+--+---+---+---+ (PCI Bus)
|
Bus IF Unit
| | |
DSP Mem Other
At the top, the PC's CPU, memory, host bridge (e.g., VIA,
Serverworks), and PCI bus of the computer. At the bottom, the Bus
Interface Unit that hooks into the DSP, card memory, and other items
on the card. For reference, see
http://focus.ti.com/docs/apps/catalog/resources/appnoteabstract.jhtml?abstractName=spru581a
[and the referenced PDF at]
http://www-s.ti.com/sc/psheets/spru581a/spru581a.pdf
and scroll down to figure 3. From what I see in this document - the
base registers appear to be able to map 4 Mbyte (w/ prefetch) or 8
Mbyte (non prefetch) regions and the location of memory is in the high
bits of the BAR. If this is the chip set used to mediate access to the
DSP / memory, your access to 16 Mbyte of card memory will have to be
managed though the BAR's. Figure 7 shows where the BAR's are
accessible from the PCI bus (PCI configuration space).
[of course - if you are using something else, please let me know]
About the VME bus, you should be able to find plenty of information. For example
http://www.vita.com/vmefaq/
at the VMEbus International Trade Association (VITA) is a pretty
extensive FAQ with references to the official specifications (e.g,
IEEE-1014), vendors and so on.
About VME addressing, I have usually used "by slot" addresses. The
applications are pretty static, once you set them up they don't change
much. The base address of the card will be handled by a configuration
file (and can be edited with any program if you move the cards).
I seem to recall that VMETRO used to have an easy to read explanation
of their card configuration on line, but I can't seem to find it any
more. They use a Universe-II chipset as the PCI to VME interface but
the addressing mode could be adapted by jumpers on the card. See
http://www.vmetro.com/midas/M-50-20.htm
for an overview of the card with a nice illustration. As mentioned
above, I have a request in to Sabtech to get the details on their
card.
The additional complexity with the VME / PMC carrier card is to
program the VME to PCI interface so you can get to the BAR's on your
card. The sequence from your CPU to your card's memory will be
something like...
CPU -> Host Bridge -> PCI Bus -> PCI / VME interface -> VME bus ->
PCI / VME interface -> PCI Bus -> (now on your card) BUS IF Unit ->
card memory
The first couple interface chips will be set up by your OS, but the
rest of the interface chips will have to be programmed by your
application or driver.
About Linux - Rubini's book is a good one. There are also plenty of
examples on line either as part of the general linux distribution or
from a private site. For example,
http://www.dolphinics.com/support/os/source.html
is an extensive set of drivers supporting Windows NT and 2000, Linux,
Lynx, Solaris, and VxWorks. They do memory mapping of the card as well
as a set of I/O control functions (e.g., send interrupt). If you are
interested, I can walk you through part of this two level driver and
point out features to improve portability.
You may also be able to do some testing from the Linux command line. For example,
setpci and getpci
allows you to modify and view the PCI configuration registers (e.g,
set up the BAR's).
As mentioned above - I'm waiting for feedback from the vendor to
confirm the operation of the cards you have chosen. Please comment on
any portion of this partial answer so I can correct it before making
the full and complete answer.
--Maniac
|
Clarification of Question by
ade3ada-ga
on
14 Jun 2004 14:10 PDT
Maniac,
Your ASCII art looks reasonable, and the description sounds good. I've
already looked at the PDF you pointed out, and yes, we're going to use
the PCI interface described there. I missed the fact that the address
space can only be up to 8 MB, but that should be enough. (Actually, I
thought that this was programmable by using an EEPROM, but that's
beside the point, so you don't need to go digging for it) However, I'm
still unsure of WHAT to put in the BAR, if anything. The high order
bits correspond to the memory space's starting address on the PCI bus,
and are set by the host's bios at initialization, so I wouldn't touch
them. The middle bits are all hardcoded zeroes and indicate the
space's size, and the low bits are for housekeeping. Nothing, though,
seems to indicate the memory space's starting address on the card's
local bus, which is the main source of my confusion.
The VME stuff looks good, and the information from Sabtech should be very useful.
As for Linux, I don't need you to walk me through writing a driver
(Thanks, though) I just want to know if it is possible to run the
device WITHOUT writing a driver for it. With older models of this card
(which I had nothing to do with) the host didn't use a custom driver.
They just memory mapped a known address range through the bus driver.
Of course, this was VME only, so they knew what the device's bus
address would be beforehand, so they could just hardcode it in the
programs on the host. I assume that we could still do this for the VME
version of the new card, but I think that the PCI version will require
a driver to communicate with the host processor. While googling,
however, I found some vague references to using Linux's /dev/mmem
filesystem to access devices without writing a driver, but they
generally indicated that this was a hack and not a very good thing to
do. So, all I really want here is some details about how to do this,
and whether it is a good idea or not. (IE, would the system be stable,
would it be worth it to just bite the bullet and write the driver,
etc.)
Thanks for the work.
Steve (ade3ada)
|
Request for Question Clarification by
maniac-ga
on
16 Jun 2004 07:06 PDT
Hello Steve,
Just a quick note to let you know I am still working on making a complete answer.
I'm still reading through the TI documentation - pretty dense but I
think I have a clue where the mapping takes place (as you noted -
perhaps not in the BAR). There are more PCI registers in the back of
the document - A rev B is now out at
http://focus.ti.com/lit/ug/spru581b/spru581b.pdf
and there appears to be a set for both slave reads / writes (prefetch
and non prefetch) and master reads / writes. Should have an answer on
this later today.
I got a copy of Sabtech's documentation - looks very straight forward
and similar to what I read from VMetro. I missed their call yesterday
but will try to talk later today and confirm my interpretation.
On /dev/mmem, that is not directly supported under Linux (at least
under 2.4 and 2.6 kernels). However, it appears that /dev/mem (major
1, minor 1) does do what you want - see
http://www.elis.ugent.be/~pfaes/pcicp.html
and
http://www.elis.ugent.be/~pfaes/altera_pci.tar.gz
for a sample program that maps memory on a PCI card and allows you to
copy to / from that memory. That should give you a good example to
work from for a (privileged) user mode application interface.
--Maniac
|
Clarification of Question by
ade3ada-ga
on
16 Jun 2004 09:51 PDT
Maniac,
Aha! Thanks for pointing me to rev B of the PCI doc. I didn't know it
was out. It looks like BAR1 is at a fixed address, and BAR0 is
relocatable by writing to the DSPP register, which lives in I/O space.
So, I need to have the host write to the DSPP before I start using the
card. Does that sound right?
The /dev/mem link was good. It seems that the main problems with this
approach are a lack of portablilty and a need to run as root (or with
priveledges). Those aren't too bad, although it doesn't look much
simpler than a small driver does. Do you know if there is a speed
penalty for using /dev/mem as opposed to a full-fledged driver?
Thanks for all your hard work, Maniac.
Steve
|
Hello Steve,
I will start the answer with a brief recap of the correct information
from the question clarification requests and then follow up with some
in depth information related to specific issues raised by the
clarifications you identified.
With respect to programming the bus interface unit, the prime
reference I used was the updated (Rev B) Reference Guide at
http://focus.ti.com/lit/ug/spru581b/spru581b.pdf
The section numbers / figures will refer to that version of the document.
Section 2 / figure 3 provides an overview into the interface provided
by the PBIN (PCI Bus Interface Unit) which also matches the ASCII art
I provided previously. The following sections go into mroe detail
until section 15 which has the details on each register used. From
that reference, we have four basic operations. Slave read / write
refers to another device initiating the read / write (e.g., CPU) and
your card is the providing / accepting the data. Master read / write
refers to the DSP as the device initiating the read / write and
another PCI device (or CPU memory) is providing / accepting the data.
Slave Writes - zero wait state transactions until the FIFO is full;
then the PCI master (e.g., CPU) will have to retry the operation.
Slave Reads - read will complete with some wait; either to get the
data directly (no prefretch) or to get the FIFO filled (prefetch).
Master Writes - writes are no wait (until the FIFO is full); the FIFO
/ PBIN will mediate bus access so the DSP can usually run at full
speed.
Master Reads - reads must be initiated / get data into the FIFO. As
with any bus read, this may imply a wait is needed. Subsequently, the
FIFO should provide data w/o wait.
For slave reads / writes and as explained in section 3 - there are
three base address registers in PCI configuration space (also shown in
figure 7). BAR0 refers to a 4 Mbyte range of memory w/ prefetch. The
DSP Page register (DSPP - 15.2.3 - I/O base +8) is used to set the 4
Mbyte region of memory used by BAR0; a 9 bit value out of a 32 bit
field. The effective address on your card is basically the 9 bits from
the DSPP followed by the 22 bits from the lower part of the PCI
address.
[odd - as I'm looking at this, I can't tell from the document if these
values are "big endian" or "little endian"; I assume its the same as
the primary DSP, but I did not find anything firm that indicates the
bit / byte order. From the bit numbering I would assume little endian
(same as Intel x86), but the document doesn't actually say which is
correct. If it doesn't appear to work right the first time, try byte
swapping the data....]
The next base address register (BAR1) refers to a 8 Mbyte region of
memory fixed at hex address 01800000 through 01FFFFFF. In this case,
the effective address on your card is 01800000 plus the lower 22 bits
of the PCI address.
The last base address register (BAR2) refers to a 16 byte region of
I/O registers. These are only available to the host side of the
interface (not available to the DSP). The offsets / contents are
described in section 15.2.
That should pretty much cover the access to your card's memory from
the host interface. You did not appear to ask about DSP initiated
transfers but I'll summarize what I noted.
DSPMA - 15.3.5 sets the word aligned DSP (not PCI) address for master
reads / writes.
PCIMA - 15.3.6 sets the word aligned PCI address for master reads /
writes. I assume this must be a PCI address reflecting the physical
location of the source / destination and not a kernel or virtual
address (both of which may be quite different).
PCIMC - 15.3.7 sets the number of bytes to transfer (count) and start
bit. I find it somewhat odd that the addresses must be word aligned,
but the count may be any value up to 65545. I expected a granularity
of four bytes (and the destination may require such a value).
There are also three "current" values reflecting the current DSP / PCI
addresses and count.
You may also have to make sure the PCI Transfer Request Control
Register (TRCTL) is set properly. This is described in 15.3.15 and
indicates a "stall" bit which allows the DSP to suspend memory
transfers, the priority, and number of outstanding requests. The
defaults allow PCI to memory access at "medium" priority with 4
requests.
From your clarification - the primary reference I provided
http://www.vita.com/vmefaq/
was OK. In my discussions with Sabtech, they confirmed my
interpretation of the implementation of their VME/PMC card and the
description I provided for that was correct (comparable to the VMetro
product). I assume you have the Sabtech Pelican VME/PMC Carrier Board
User's Guide which appears to have good explanations of the operation
/ programming of the card. Sabtech also said they had sample code for
VXWorks if that would help. As a result I won't expand further on the
VME solution unless there is something specific you are having a
problem with.
On interfacing with Linux, you basically have the following methods available:
[1] use existing applications such as setpci / getpci to examine the
PCI configuration registers. Note that a privileged account (e.g.,
root) is needed to modify PCI registers and to read beyond the initial
section of the PCI registers [safety of operation reasons].
[2] opening /dev/mem and using a memory map (e.g., mmap) to gain
access to PCI addresses. This is not very portable (for example, x86
on Linux only) and also requires a privileged account (e.g,. root) to
succeed since unrestricted access to physical memory / the PCI bus is
either dangerous to system operation or a serious security hole. The
example I found at
http://www.elis.ugent.be/~pfaes/pcicp.html
implements this kind of operation. You will probably need to ask some
customers (current or prospective) to see if they would agree to such
a solution. For example - if they do a real time application, they
likely run as root already to get real time priority / lock the
application into memory so this is not a problem. Others may not have
such needs and would not have such access (e.g., a shared system). You
asked if there is a speed penalty for doing this - as far as I can
tell, there is no speed penalty since the mmap operation sets up the
CPU page tables to map your virtual addresses (within a range) to
address the PCI bus directly. This is basically the same as what your
device driver will do if it provides such a solution. Note that PCI
memory references are MUCH SLOWER than accesses to local memory and
that read accesses will often be slower than write accesses. In some
ways it is better to copy the data from the PCI bus into local memory,
manipulate it there, and then write a large block to the PCI bus to
optimize application speed [applies to both #2 and #3].
[3] creating a simple device driver and providing user access to that
driver. This requires root access to install the driver, create the
special files under /dev, and modify the system start scripts to load
the driver at startup. Once done, a "normal" user can then have an
application that works with the driver to get the work done in a more
safe manner. As noted at Dolphin's site
http://www.dolphinics.com/support/os/source.html
it is feasible to create a device driver suitable for use on both
Linux and Windows (as well as several other systems). This involves
more up front work than the other solutions but should be easier for
your users in the long run.
It may be best at this point to use [1] and [2] to do initial check
out of the card and maybe some of the diagnostics you provide to your
customer to do checkout and fault isolation. As you become more
familiar with how the card works and have time then do [3] to provide
a more complete solution to your users.
That should cover all the issues identified in your question. Don't
hesitate to make a clarification request if some part of the answer
appears incorrect or incomplete. I would also be glad to expand on
some related topic if needed.
Good luck with your work and I hope your card turns out to be a good
product for your customers (and your company).
--Maniac |
