Skip to content
Snippets Groups Projects
Commit f7eb9ece authored by cvs2svn's avatar cvs2svn
Browse files

This commit was manufactured by cvs2svn to create branch

'unlabeled-1.8.26.1.2'.
parent 34c69e33
No related branches found
No related tags found
No related merge requests found
Showing
with 0 additions and 3483 deletions
Kernelenv
Makefile
autoMakefile
autoMakefile.in
aclocal.m4
autom4te.cache
config.log
config.status
configure
.*.cmd
.depend
This diff is collapsed.
EXTRA_CFLAGS := -Ifs/lustre/include -Ifs/lustre/lnet/include
# lnet/utils/debug.c wants <linux/version.h> from userspace. sigh.
HOSTCFLAGS := -I@LINUX@/include $(EXTRA_CFLAGS)
LIBREADLINE := @LIBREADLINE@
# 2.5's makefiles aren't nice to cross dir libraries in host programs
PTLCTLOBJS := debug.o l_ioctl.o parser.o portals.o
EXTRA_CFLAGS := -Ifs/lustre/include -Ifs/lustre/lnet/include
HOSTCFLAGS := $(EXTRA_CFLAGS)
# the kernel doesn't want us to build archives for host binaries :/
PTLCTLOBJS := debug.o l_ioctl.o parser.o portals.o
Each file in this distribution should contain a header stating the
copyright owner(s), and the licensing terms for that module. Some
files are not eligible for copyright protection, and contain neither.
All files in this subtree are licensed under the terms and conditions
of the GNU General Public License version 2.
Reproduced below is the GPL v2, and Linus's clarifying statement from
the Linux kernel source code:
----------------------------------------
NOTE! This copyright does *not* cover user programs that use kernel
services by normal system calls - this is merely considered normal use
of the kernel, and does *not* fall under the heading of "derived work".
Also note that the GPL below is copyrighted by the Free Software
Foundation, but the instance of code that it refers to (the Linux
kernel) is copyrighted by me and others who actually wrote it.
Linus Torvalds
----------------------------------------
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) 19yy <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) 19yy name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.
subdir-m += libcfs
lnet-subdirs += lnet
lnet-subdirs += klnds
lnet-subdirs += selftest
subdir-m += $(lnet-subdirs)
@INCLUDE_RULES@
# Copyright (C) 2001 Cluster File Systems, Inc.
#
# This code is issued under the GNU General Public License.
# See the file COPYING in this distribution
SUBDIRS = libcfs lnet klnds ulnds selftest doc utils include \
autoconf
sources:
$(MAKE) sources -C libcfs
Makefile
Makefile.in
EXTRA_DIST := lustre-lnet.m4
This diff is collapsed.
Makefile
Makefile.in
*.eps
*.pdf
In this document I will try to draw the data structures and how they
interrelate in the Portals 3 reference implementation. It is probably
best shown with a drawing, so there may be an additional xfig or
Postscript figure.
MEMORY POOLS:
------------
First, a digression on memory allocation in the library. As mentioned
in the NAL Writer's Guide, the library does not link against any
standard C libraries and as such is unable to dynamically allocate
memory on its own. It requires that the NAL implement a method
for allocation that is appropriate for the protection domain in
which the library lives. This is only called when a network
interface is initialized to allocate the Portals object pools.
These pools are preallocate blocks of objects that the library
can rapidly make active and manage with a minimum of overhead.
It is also cuts down on overhead for setting up structures
since the NAL->malloc() callback does not need to be called
for each object.
The objects are maintained on a per-object type singly linked free
list and contain a pointer to the next free object. This pointer
is NULL if the object is not on the free list and is non-zero
if it is on the list. The special sentinal value of 0xDEADBEEF
is used to mark the end of the free list since NULL could
indicate that the last object in the list is not free.
When one of the lib_*_alloc() functions is called, the library
returns the head of the free list and advances the head pointer
to the next item on the list. The special case of 0xDEADBEEF is
checked and a NULL pointer is returned if there are no more
objects of this type available. The lib_*_free() functions
are even simpler -- check to ensure that the object is not already
free, set its next pointer to the current head and then set
the head to be this newly freed object.
Since C does not have templates, I did the next best thing and wrote
the memory pool allocation code as a macro that expands based on the
type of the argument. The mk_alloc(T) macro expands to
write the _lib_T_alloc() and lib_T_free() functions.
It requires that the object have a pointer of the type T named
"next_free". There are also functions that map _lib_T_alloc()
to lib_T_alloc() so that the library can add some extra
functionality to the T constructor.
LINKED LISTS:
------------
Many of the active Portals objects are stored in doubly linked lists
when they are active. These are always implemented with the pointer
to the next object and a pointer to the next pointer of the
previous object. This avoids the "dummy head" object or
special cases for inserting at the beginning or end of the list.
The pointer manipulations are a little hairy at times, but
I hope that they are understandable.
The actual linked list code is implemented as macros in <lib-p30.h>,
although the object has to know about
# Copyright (C) 2001 Cluster File Systems, Inc.
#
# This code is issued under the GNU General Public License.
# See the file COPYING in this distribution
LYX2PDF = lyx --export pdf
LYX2TXT = lyx --export text
LYX2HTML = lyx --export html
SUFFIXES = .lin .lyx .pdf .sgml .html .txt .fig .eps
if DOC
DOCS = portals3.pdf
else
DOCS =
endif
IMAGES = file.eps flow_new.eps get.eps mpi.eps portals.eps put.eps
LYXFILES= portals3.lyx
MAINTAINERCLEANFILES = $(IMAGES) $(DOCS) $(GENERATED)
GENERATED =
EXTRA_DIST = $(DOCS) $(IMAGES) $(LYXFILES)
all: $(DOCS)
# update date and version in document
date := $(shell date +%x)
tag := $(shell echo '$$Name: $$' | sed -e 's/^\$$Na''me: *\$$$$/HEAD/; s/^\$$Na''me: \(.*\) \$$$$/\1/')
addversion = sed -e 's|@T''AG@|$(tag)|g; s|@VER''SION@|$(VERSION)|g; s|@DA''TE@|$(date)|g'
# Regenerate when the $(VERSION) or $Name: $ changes.
.INTERMEDIATE: $(GENERATED)
$(GENERATED) : %.lyx: %.lin Makefile
$(addversion) $< > $@
.lyx.pdf:
@$(LYX2PDF) $< || printf "\n*** Warning: not creating PDF docs; install lyx to rectify this\n"
.lyx.txt:
@$(LYX2TXT) $< || printf "\n*** Warning: not creating text docs; install lyx to rectify this\n"
.lyx.html:
@$(LYX2HTML) $< || printf "\n*** Warning: not creating HTML docs; install lyx to rectify this\n"
.fig.eps:
-fig2dev -L eps $< > $@
portals3.pdf portals3.txt portals3.html: $(IMAGES) portals3.lyx
syncweb: portals3.pdf
# cp lustre.pdf /usr/src/www/content/lustre/docs/lustre.pdf
# ( cd /usr/src/www ; make lustre ; make synclustre )
This documents the life cycle of message as it arrives and is handled by
a basic async, packetized NAL. There are four types of messages that have
slightly different life cycles, so they are addressed independently.
Put request
-----------
1. NAL notices that there is a incoming message header on the network
and reads an ptl_hdr_t in from the wire.
2. It may store additional NAL specific data that provides context
for this event in a void* that it will interpret in some fashion
later.
3. The NAL calls lib_parse() with a pointer to the header and its
private data structure.
4. The library decodes the header and may build a message state
object that describes the event to be written and the ACK to be
sent, if any. It then calls nal->recv() with the private data
that the NAL passed in, a pointer to the message state object
and a translated user address.
The NAL will have been given a chance to pretranslate
all user addresses when the buffers are created. This
process is described in the NAL-HOWTO.
5. The NAL should restore what ever context it required from the
private data pointer, begin receiving the bytes and possibly store
some extra state of its own. It should return at this point.
Get request
-----------
1. As with a Put, the NAL notices the incoming message header and
passes it to lib_parse().
2. The library decodes the header and calls nal->recv() with a
zero byte length, offset and destination to instruct it to clean
up the wire after reading the header. The private data will
be passed in as well, allowing the NAL to retrieve any state
or context that it requires.
3. The library may build a message state object to possibly
write an event log or invalidate a memory region.
4. The library will build a ptl_msg_t header that specifies the
Portals protocol information for delivery at the remote end.
5. The library calls nal->send() with the pre-built header,
the optional message state object, the four part address
component, a translated user pointer + offset, and some
other things.
6. The NAL is to put the header on the wire or copy it at
this point (since it off the stack). It should store some
amount of state about its current position in the message and
the destination address.
7. And then return to the library.
Reply request
-------------
1. Starting at "The library decodes the header..."
2. The library decodes the header and calls nal->recv()
to bring in the rest of the message. Flow continues in
exactly the same fashion as with all other receives.
Ack request
-----------
1. The library decodes the header, builds the appropriate data
structures for the event in a message state object and calls nal->recv()
with a zero byte length, etc.
Packet arrival
--------------
1. The NAL should notice the arrival of a packet, retrieve whatever
state it needs from the message ID or other NAL specific header data
and place the data bytes directly into the user address that were
given to nal->recv().
How this happens is outside the scope of the Portals library
and soley determined by the NAL...
2. If this is the last packet in a message, the NAL should retrieve
the lib_msg_t *cookie that it was given in the call to nal->recv()
and pass it to lib_finalize(). lib_finalize() may call nal->send()
to send an ACK, nal->write() to record an entry in the event log,
nal->invalidate() to unregister a region of memory or do nothing at all.
3. It should then clean up any remaining NAL specific state about
the message and go back into the main loop.
Outgoing packets
----------------
1. When the NAL has pending output, it should put the packets on
the wire wrapped with whatever implementation specified wrappers.
2. Once it has output all the packets of a message it should
call lib_finalize() with the message state object that was
handed to nal->send(). This will allows the library to clean
up its state regarding the message and write any pending event
entries.
This document is a first attempt at describing how to write a NAL
for the Portals 3 library. It also defines the library architecture
and the abstraction of protection domains.
First, an overview of the architecture:
Application
----|----+--------
|
API === NAL (User space)
|
---------+---|-----
|
LIB === NAL (Library space)
|
---------+---|-----
Physical wire (NIC space)
Application
API
API-side NAL
------------
LIB-side NAL
LIB
LIB-side NAL
wire
Communication is through the indicated paths via well defined
interfaces. The API and LIB portions are written to be portable
across platforms and do not depend on the network interface.
Communcation between the application and the API code is
defined in the Portals 3 API specification. This is the
user-visible portion of the interface and should be the most
stable.
API-side NAL:
------------
The user space NAL needs to implement only a few functions
that are stored in a nal_t data structure and called by the
API-side library:
int forward( nal_t *nal,
int index,
void *args,
size_t arg_len,
void *ret,
size_t ret_len
);
Most of the data structures in the portals library are held in
the LIB section of the code, so it is necessary to forward API
calls across the protection domain to the library. This is
handled by the NAL's forward method. Once the argument and return
blocks are on the remote side the NAL should call lib_dispatch()
to invoke the appropriate API function.
int validate( nal_t *nal,
void *base,
size_t extent,
void **trans_base,
void **trans_data
);
The validate method provides a means for the NAL to prevalidate
and possibly pretranslate user addresses into a form suitable
for fast use by the network card or kernel module. The trans_base
pointer will be used by the library everytime it needs to
refer to the block of memory. The trans_data result is a
cookie that will be handed to the NAL along with the trans_base.
The library never performs calculations on the trans_base value;
it only computes offsets that are then handed to the NAL.
int shutdown( nal_t *nal, int interface );
Brings down the network interface. The remote NAL side should
call lib_fini() to bring down the library side of the network.
void yield( nal_t *nal );
This allows the user application to gracefully give up the processor
while busy waiting. Performance critical applications may not
want to take the time to call this function, so it should be an
option to the PtlEQWait call. Right now it is not implemented as such.
Lastly, the NAL must implement a function named PTL_IFACE_*, where
* is the name of the NAL such as PTL_IFACE_IP or PTL_IFACE_MYR.
This initialization function is to set up communication with the
library-side NAL, which should call lib_init() to bring up the
network interface.
LIB-side NAL:
------------
On the library-side, the NAL has much more responsibility. It
is responsible for calling lib_dispatch() on behalf of the user,
it is also responsible for bringing packets off the wire and
pushing bits out. As on the user side, the methods are stored
in a nal_cb_t structure that is defined on a per network
interface basis.
The calls to lib_dispatch() need to be examined. The prototype:
void lib_dispatch(
nal_cb_t *nal,
void *private,
int index,
void *arg_block,
void *ret_block
);
has two complications. The private field is a NAL-specific
value that will be passed to any callbacks produced as a result
of this API call. Kernel module implementations may use this
for task structures, or perhaps network card data. It is ignored
by the library.
Secondly, the arg_block and ret_block must be in the same protection
domain as the library. The NAL's two halves must communicate the
sizes and perform the copies. After the call, the buffer pointed
to by ret_block will be filled in and should be copied back to
the user space. How this is to be done is NAL specific.
int lib_parse(
nal_cb_t *nal,
ptl_hdr_t *hdr,
void *private
);
This is the only other entry point into the library from the NAL.
When the NAL detects an incoming message on the wire it should read
sizeof(ptl_hdr_t) bytes and pass a pointer to the header to
lib_parse(). It may set private to be anything that it needs to
tie the incoming message to callbacks that are made as a result
of this event.
The method calls are:
int (*send)(
nal_cb_t *nal,
void *private,
lib_msg_t *cookie,
ptl_hdr_t *hdr,
int nid,
int pid,
int gid,
int rid,
user_ptr trans_base,
user_ptr trans_data,
size_t offset,
size_t len
);
This is a tricky function -- it must support async output
of messages as well as properly syncronized event log writing.
The private field is the same that was passed into lib_dispatch()
or lib_parse() and may be used to tie this call to the event
that initiated the entry to the library.
The cookie is a pointer to a library private value that must
be passed to lib_finalize() once the message has been completely
sent. It should not be examined by the NAL for any meaning.
The four ID fields are passed in, although some implementations
may not use all of them.
The single base pointer has been replaced with the translated
address that the API NAL generated in the api_nal->validate()
call. The trans_data is unchanged and the offset is in bytes.
int (*recv)(
nal_cb_t *nal,
void *private,
lib_msg_t *cookie,
user_ptr trans_base,
user_ptr trans_data,
size_t offset,
size_t mlen,
size_t rlen
);
This callback will only be called in response to lib_parse().
The cookie, trans_addr and trans_data are as discussed in send().
The NAL should read mlen bytes from the wire, deposit them into
trans_base + offset and then discard (rlen - mlen) bytes.
Once the entire message has been received the NAL should call
lib_finalize() with the lib_msg_t *cookie.
The special arguments of base=NULL, data=NULL, offset=0, mlen=0, rlen=0
is used to indicate that the NAL should clean up the wire. This could
be implemented as a blocking call, although having it return as quickly
as possible is desirable.
int (*write)(
nal_cb_t *nal,
void *private,
user_ptr trans_addr,
user_ptr trans_data,
size_t offset,
void *src_addr,
size_t len
);
This is essentially a cross-protection domain memcpy(). The user address
has been pretranslated by the api_nal->translate() call.
void *(*malloc)(
nal_cb_t *nal,
size_t len
);
void (*free)(
nal_cb_t *nal,
void *buf
);
Since the NAL may be in a non-standard hosted environment it can
not call malloc(). This allows the library side NAL to implement
the system specific malloc(). In the current reference implementation
the libary only calls nal->malloc() when the network interface is
initialized and then calls free when it is brought down. The library
maintains its own pool of objects for allocation so only one call to
malloc is made per object type.
void (*invalidate)(
nal_cb_t *nal,
user_ptr trans_base,
user_ptr trans_data,
size_t extent
);
User addresses are validated/translated at the user-level API NAL
method, which is likely to push them to this level. Meanwhile,
the library NAL will be notified when the library no longer
needs the buffer. Overlapped buffers are not detected by the
library, so the NAL should ref count each page involved.
Unfortunately we have a few bugs when the invalidate method is
called. It is still in progress...
void (*printf)(
nal_cb_t *nal,
const char *fmt,
...
);
As with malloc(), the library does not have any way to do printf
or printk. It is not necessary for the NAL to implement the this
call, although it will make debugging difficult.
void (*cli)(
nal_cb_t *nal,
unsigned long *flags
);
void (*sti)(
nal_cb_t *nal,
unsigned long *flags
);
These are used by the library to mark critical sections.
int (*gidrid2nidpid)(
nal_cb_t *nal,
ptl_id_t gid,
ptl_id_t rid,
ptl_id_t *nid,
ptl_id_t *pid
);
int (*nidpid2gidrid)(
nal_cb_t *nal,
ptl_id_t nid,
ptl_id_t pid,
ptl_id_t *gid,
ptl_id_t *rid
);
Rolf added these. I haven't looked at how they have to work yet.
#FIG 3.2
Landscape
Center
Inches
Letter
100.00
Single
-2
1200 2
6 1200 750 1650 1050
2 4 0 1 0 7 100 0 -1 0.000 0 0 7 0 0 5
1650 1050 1650 750 1200 750 1200 1050 1650 1050
4 1 0 100 0 0 10 0.0000 0 105 240 1425 952 FS0\001
-6
6 1200 2325 1650 2625
2 4 0 1 0 7 100 0 -1 0.000 0 0 7 0 0 5
1650 2625 1650 2325 1200 2325 1200 2625 1650 2625
4 1 0 100 0 0 10 0.0000 0 105 240 1425 2527 FS3\001
-6
6 1200 1800 1650 2100
2 4 0 1 0 7 100 0 -1 0.000 0 0 7 0 0 5
1650 2100 1650 1800 1200 1800 1200 2100 1650 2100
4 1 0 100 0 0 10 0.0000 0 105 240 1425 2002 FS2\001
-6
6 1200 1275 1650 1575
2 4 0 1 0 7 100 0 -1 0.000 0 0 7 0 0 5
1650 1575 1650 1275 1200 1275 1200 1575 1650 1575
4 1 0 100 0 0 10 0.0000 0 105 240 1425 1477 FS1\001
-6
6 450 750 900 1200
5 1 0 1 0 7 100 0 20 0.000 0 1 0 0 675.000 750.000 450 1050 675 1125 900 1050
1 2 0 1 0 7 100 0 20 0.000 1 0.0000 675 825 225 75 450 900 900 750
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
450 825 450 1050
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 1050 900 825
-6
6 450 2325 900 2775
5 1 0 1 0 7 100 0 20 0.000 0 1 0 0 675.000 2325.000 450 2625 675 2700 900 2625
1 2 0 1 0 7 100 0 20 0.000 1 0.0000 675 2400 225 75 450 2475 900 2325
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
450 2400 450 2625
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 2625 900 2400
-6
6 450 1800 900 2250
5 1 0 1 0 7 100 0 20 0.000 0 1 0 0 675.000 1800.000 450 2100 675 2175 900 2100
1 2 0 1 0 7 100 0 20 0.000 1 0.0000 675 1875 225 75 450 1950 900 1800
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
450 1875 450 2100
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 2100 900 1875
-6
6 450 1275 900 1725
5 1 0 1 0 7 100 0 20 0.000 0 1 0 0 675.000 1275.000 450 1575 675 1650 900 1575
1 2 0 1 0 7 100 0 20 0.000 1 0.0000 675 1350 225 75 450 1425 900 1275
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
450 1350 450 1575
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 1575 900 1350
-6
6 2250 750 3450 2625
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 2
2550 1200 3150 1200
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 2
2550 1500 3150 1500
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 2
2550 1800 3150 1800
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 2
2550 2100 3150 2100
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
2550 975 3150 975 3150 2625 2550 2625 2550 975
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 2
2550 2400 3150 2400
4 1 0 100 0 0 10 0.0000 0 135 1185 2850 900 Application Buffer\001
-6
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 2400 2550 1350
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 1875 2550 1050
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 1425 2550 1950
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 900 2550 1650
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 900 1200 900
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 1425 1200 1425
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 1950 1200 1950
2 1 0 1 0 7 100 0 20 0.000 0 0 -1 0 0 2
900 2475 1200 2475
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 2025 2550 2250
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 1 2
0 0 1.00 60.00 120.00
0 0 1.00 60.00 120.00
1650 2550 2550 2475
2 4 0 1 0 7 100 0 -1 0.000 0 0 7 0 0 5
1875 2850 1875 600 225 600 225 2850 1875 2850
4 1 0 100 0 0 10 0.0000 0 105 1215 1050 525 Parallel File Server\001
#FIG 3.2
Landscape
Center
Inches
Letter
100.00
Single
-2
1200 2
6 525 2175 1575 2925
6 675 2287 1425 2812
4 1 0 50 0 0 10 0.0000 4 105 255 1050 2437 MD\001
4 1 0 50 0 0 10 0.0000 4 105 645 1050 2587 Exists and\001
4 1 0 50 0 0 10 0.0000 4 135 555 1050 2737 Accepts?\001
-6
2 3 0 1 0 7 100 0 -1 0.000 0 0 0 0 0 5
1575 2550 1050 2175 525 2550 1050 2925 1575 2550
-6
6 3450 1275 4350 1725
6 3600 1312 4200 1687
4 1 0 100 0 0 10 0.0000 0 135 525 3900 1612 Message\001
4 1 0 100 0 0 10 0.0000 0 105 465 3900 1462 Discard\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
3450 1275 4350 1275 4350 1725 3450 1725 3450 1275
-6
6 4650 1275 5550 1725
6 4725 1312 5475 1687
4 1 0 100 0 0 10 0.0000 0 135 735 5100 1612 Drop Count\001
4 1 0 100 0 0 10 0.0000 0 105 630 5100 1462 Increment\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
4650 1275 5550 1275 5550 1725 4650 1725 4650 1275
-6
6 1350 525 2250 975
6 1350 562 2250 937
4 1 0 100 0 0 10 0.0000 0 135 795 1800 862 Match Entry\001
4 1 0 100 0 0 10 0.0000 0 105 585 1800 712 Get Next\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
1350 525 2250 525 2250 975 1350 975 1350 525
-6
6 525 1125 1575 1875
2 3 0 1 0 7 100 0 -1 0.000 0 0 0 0 0 5
1575 1500 1050 1125 525 1500 1050 1875 1575 1500
4 1 0 100 0 0 10 0.0000 0 105 465 1049 1552 Match?\001
-6
6 2340 1237 2940 1687
6 2340 1237 2940 1687
4 1 0 100 0 0 10 0.0000 0 105 345 2640 1387 More\001
4 1 0 100 0 0 10 0.0000 0 105 405 2640 1537 Match\001
4 1 0 100 0 0 10 0.0000 0 105 510 2640 1687 Entries?\001
-6
-6
6 525 3225 1575 3975
6 675 3375 1425 3750
4 1 0 50 0 0 10 0.0000 4 105 255 1050 3525 MD\001
4 1 0 50 0 0 10 0.0000 4 105 615 1050 3720 has room?\001
-6
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5
525 3600 1050 3225 1575 3600 1050 3975 525 3600
-6
6 3300 3375 4350 3825
6 3300 3412 4350 3787
4 1 0 50 0 0 10 0.0000 4 105 735 3825 3562 Unlink MD\001
4 1 0 50 0 0 10 0.0000 4 135 945 3825 3712 & Match Entry\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
3300 3375 4350 3375 4350 3825 3300 3825 3300 3375
-6
6 1950 3225 3000 3975
6 2250 3450 2700 3750
4 1 0 50 0 0 10 0.0000 4 105 450 2475 3600 Unlink\001
4 1 0 50 0 0 10 0.0000 4 105 315 2475 3750 full?\001
-6
2 3 0 1 0 7 100 0 -1 0.000 0 0 0 0 0 5
3000 3600 2475 3225 1950 3600 2475 3975 3000 3600
-6
6 3150 4500 4200 4950
6 3150 4537 4200 4912
4 1 0 50 0 0 10 0.0000 4 105 735 3675 4687 Unlink MD\001
4 1 0 50 0 0 10 0.0000 4 135 945 3675 4837 & Match Entry\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
3150 4500 4200 4500 4200 4950 3150 4950 3150 4500
-6
6 600 4500 1500 4950
6 675 4537 1425 4912
4 1 0 50 0 0 10 0.0000 4 135 615 1050 4837 Operation\001
4 1 0 50 0 0 10 0.0000 4 105 525 1050 4687 Perform\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
600 4500 1500 4500 1500 4950 600 4950 600 4500
-6
6 4650 4350 5700 5100
6 4950 4537 5400 4912
6 4950 4537 5400 4912
4 1 0 50 0 0 10 0.0000 4 135 435 5175 4837 Queue?\001
4 1 0 50 0 0 10 0.0000 4 105 360 5175 4687 Event\001
-6
-6
2 3 0 1 0 7 100 0 -1 0.000 0 0 0 0 0 5
5700 4725 5175 4350 4650 4725 5175 5100 5700 4725
-6
6 6000 4500 6900 4950
6 6225 4575 6675 4875
4 1 0 50 0 0 10 0.0000 4 105 360 6450 4875 Event\001
4 1 0 50 0 0 10 0.0000 4 105 435 6450 4725 Record\001
-6
2 2 0 1 0 7 100 0 -1 0.000 0 0 -1 0 0 5
6000 4500 6900 4500 6900 4950 6000 4950 6000 4500
-6
6 1800 4350 2850 5100
6 2100 4575 2550 4875
4 1 0 50 0 0 10 0.0000 4 105 450 2325 4725 Unlink\001
4 1 0 50 0 0 10 0.0000 4 105 450 2325 4875 thresh?\001
-6
2 3 0 1 0 7 100 0 -1 0.000 0 0 0 0 0 5
2850 4725 2325 4350 1800 4725 2325 5100 2850 4725
-6
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1050 1875 1050 2175
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1575 1500 2100 1500
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1050 450 1050 1125
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1350 750 1050 750
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1050 2925 1050 3225
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
3150 1500 3450 1500
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
4350 1500 4650 1500
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 5
2100 1500 2625 1125 3150 1500 2625 1875 2100 1500
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1575 3600 1950 3600
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1050 3975 1050 4500
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
3000 3600 3300 3600
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
1500 4725 1800 4725
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
5700 4725 6000 4725
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
2850 4725 3150 4725
2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
4200 4725 4650 4725
2 1 0 1 0 7 100 0 -1 0.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
6900 4725 7950 4725
3 0 0 1 0 7 100 0 -1 0.000 0 1 0 5
0 0 1.00 60.00 120.00
1575 2550 1650 2550 1800 2550 1800 2400 1800 1500
0.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 100 0 -1 0.000 0 0 1 5
0 0 1.00 60.00 120.00
2250 750 2475 750 2625 750 2625 900 2625 1125
0.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 100 0 -1 0.000 0 0 1 5
0 0 1.00 60.00 120.00
7500 4725 7500 1650 7500 1500 7350 1500 5550 1500
0.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 50 0 -1 0.000 0 1 0 5
0 0 1.00 60.00 120.00
2475 3225 2475 2400 2475 2250 2325 2250 1800 2250
0.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 50 0 -1 0.000 0 1 0 5
0 0 1.00 60.00 120.00
3825 3375 3825 2175 3825 2025 3675 2025 1800 2025
0.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8
0 0 1.00 60.00 120.00
2325 4350 2325 4275 2325 4125 2475 4125 4275 4125 4425 4125
4425 4275 4425 4725
0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000
3 0 0 1 0 7 50 0 -1 0.000 0 1 0 8
0 0 1.00 60.00 120.00
5175 4350 5175 4275 5175 4125 5325 4125 7125 4125 7275 4125
7275 4275 7275 4725
0.000 1.000 1.000 1.000 1.000 1.000 1.000 0.000
4 1 0 100 0 0 10 0.0000 0 75 150 1575 1425 no\001
4 1 0 100 0 0 10 0.0000 0 135 360 825 525 Entry\001
4 1 0 100 0 0 10 0.0000 0 75 150 1575 2475 no\001
4 1 0 100 0 0 10 0.0000 0 105 195 1200 1950 yes\001
4 1 0 100 0 0 10 0.0000 0 105 195 1200 3000 yes\001
4 1 0 100 0 0 10 0.0000 0 105 195 2775 1050 yes\001
4 1 0 100 0 0 10 0.0000 0 75 150 3225 1425 no\001
4 1 0 100 0 0 10 0.0000 0 75 150 1650 3525 no\001
4 1 0 100 0 0 10 0.0000 0 105 195 1200 4050 yes\001
4 1 0 100 0 0 10 0.0000 0 105 195 3150 3525 yes\001
4 1 0 100 0 0 10 0.0000 0 75 150 2625 3150 no\001
4 1 0 100 0 0 10 0.0000 0 105 195 3000 4650 yes\001
4 1 0 100 0 0 10 0.0000 0 105 195 5850 4650 yes\001
4 1 0 100 0 0 10 0.0000 0 75 150 2475 4275 no\001
4 1 0 100 0 0 10 0.0000 0 75 150 5325 4275 no\001
4 1 0 50 0 0 10 0.0000 4 105 285 7800 4650 Exit\001
#FIG 3.2
Landscape
Center
Inches
Letter
100.00
Single
-2
1200 2
6 2775 900 3525 1200
4 0 0 100 0 0 10 0.0000 0 105 720 2775 1200 Translation\001
4 0 0 100 0 0 10 0.0000 0 105 405 2850 1050 Portal\001
-6
6 1350 1725 2175 2025
4 0 0 100 0 0 10 0.0000 0 105 825 1350 2025 Transmission\001
4 0 0 100 0 0 10 0.0000 0 105 285 1620 1875 Data\001
-6
2 1 0 1 0 7 100 0 -1 4.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
900 525 2700 750
2 1 0 1 0 7 100 0 -1 4.000 0 0 -1 1 0 2
0 0 1.00 60.00 120.00
2700 825 2700 1275
2 1 0 1 0 7 100 0 -1 3.000 0 0 7 1 0 2
0 0 1.00 60.00 120.00
2700 1350 900 1950
2 2 0 1 0 7 100 0 -1 4.000 0 0 7 0 0 5
2400 300 3600 300 3600 2250 2400 2250 2400 300
2 2 0 1 0 7 100 0 -1 4.000 0 0 7 0 0 5
0 300 1200 300 1200 2250 0 2250 0 300
4 1 0 100 0 0 10 0.0000 4 135 495 1800 825 Request\001
4 1 0 100 0 0 10 0.0000 0 105 540 600 525 Initiator\001
4 1 0 100 0 0 10 0.0000 0 135 405 3000 525 Target\001
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment